Skip to content

Too strong config validation for clustering option #290

@zachary-povey

Description

@zachary-povey

I may have misunderstood, but in com.wepay.kafka.connect.bigquery.config.BigQuerySinkTaskConfig line 243 it seems that we are asserting that if timestampPartitionFieldName isn't set and bigQueryPartitionDecorator is set to false then the resulting table won't be partitioned (and thus we cannot cluster).

I don't believe this is the case, we will still be partitioned using the _PARTITIONTIME pseudo-column, however the messages will be assigned a partition by BigQuery based on when the message was received there - right?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions