Skip to content

Convert io.debezium.time.MicroTimestamp to DATETIME  #305

@LarsKlingen

Description

@LarsKlingen

For legacy reasons I would like to have DATETIME formau for columns that come as MicroTimestamp (We use Debezium for postgressql that converts timestamps to MicroTimestamp). The Sink automatically converts to TIMESTAMP format.
I tried creating the table first with type DATETIME for the columns to circumentvent this, but then I got the error:

Field updated_at has changed type from DATETIME to TIMESTAMP; See logs for more detail
	at com.wepay.kafka.connect.bigquery.write.batch.KCBQThreadPoolExecutor.maybeThrowEncounteredErrors(KCBQThreadPoolExecutor.java:108)
	at com.wepay.kafka.connect.bigquery.BigQuerySinkTask.put(BigQuerySinkTask.java:233)
	at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:582)
	... 10 more

Is there a way to save columns as DATETIME on BQ? I also considered using a view to convert to DATETIME, but then we lose the parittioning property on these columns. Thanks!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions