_version column is filled with error string

[Fluder-Paradyne]

For mysql without gti enabled, the version column is filled with this error.

<failed to load>
java.sql.SQLException: Method: getLong("_version") encountered an exception.
	at com.clickhouse.jdbc.internal.ExceptionUtils.toSqlState(ExceptionUtils.java:74)
	at com.clickhouse.jdbc.ResultSetImpl.getLong(ResultSetImpl.java:352)
	at com.clickhouse.jdbc.ResultSetImpl.getLong(ResultSetImpl.java:207)
	in CHouseJdbcHelper.tryGetObject(CHouseJdbcHelper.java:157)
Caused by: java.lang.ArithmeticException: integer overflow: 18446744073709551615 cannot be presented as long
	at com.clickhouse.client.api.data_formats.internal.NumberConverter.toLong(NumberConverter.java:87)
	at com.clickhouse.client.api.data_formats.internal.AbstractBinaryFormatReader.readNumberValue(AbstractBinaryFormatReader.java:350)
	at com.clickhouse.client.api.data_formats.internal.AbstractBinaryFormatReader.getLong(AbstractBinaryFormatReader.java:374)
	at com.clickhouse.jdbc.ResultSetImpl.getLong(ResultSetImpl.java:346)
	... 23 more

[aadant]

@Fluder-Paradyne this case is already covered. Debezium returns a list of ordered events. from each start of second, a counter is incremented which makes it replayable. see #523
I wonder what caused this situation in the first place. can you provide more details on how to reproduce ?

[Fluder-Paradyne]

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnector
metadata:
  name: meeting-mysql-cdc-v1
  labels:
    strimzi.io/cluster: my-kafka-connect
spec:
  class: io.debezium.connector.mysql.MySqlConnector
  tasksMax: 1
  config:
    # Snapshot settings
    snapshot.mode: initial
    snapshot.locking.mode: none
    snapshot.delay.ms: "10000"
    include.schema.changes: "true"

    # Database connection settings
    database.hostname: <REDACTED>
    database.port: "3306"
    database.user: <REDACTED>
    database.password: <REDACTED>

    # Unique identifier for this connector
    database.server.id: "10002"
    topic.prefix: meeting_orchestrator

    # Database filtering
    database.include.list: meeting_production

    # MySQL connection settings (Aurora MySQL)
    database.allowPublicKeyRetrieval: "true"
    database.useSSL: "false"

    # Schema history configuration
    schema.history.internal.kafka.bootstrap.servers: kafka-bootstrap:9092
    schema.history.internal.kafka.topic: schema-changes.meeting_orchestrator
    schema.history.internal.recovery.mode: "false"
    schema.history.internal.skip.unparseable.ddl: "true"

    # Topic creation settings
    topic.creation.default.replication.factor: "2"
    topic.creation.default.partitions: "3"

    # Error handling
    errors.tolerance: all
    errors.log.enable: "true"
    errors.log.include.messages: "true"

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnector
metadata:
  name: meeting-clickhouse-sink-connector
  labels:
    strimzi.io/cluster: my-kafka-connect
spec:
  class: com.altinity.clickhouse.sink.connector.ClickHouseSinkConnector
  tasksMax: 10
  config:
    # Kafka topics from Debezium MySQL connector
    topics.regex: "meeting_orchestrator\\.meeting_production\\..*"

    # ClickHouse connection settings
    clickhouse.server.url: <REDACTED>
    clickhouse.server.port: "8123"
    clickhouse.server.user: <REDACTED>
    clickhouse.server.password: <REDACTED>
    clickhouse.server.database: meeting_production

    # Buffer and batch settings
    buffer.count: "1000"
    buffer.flush.time.ms: "1000"
    buffer.flush.timeout.ms: "30000"
    buffer.max.records: "10000"
    offset.flush.interval.ms: "5000"

    # Thread pool settings
    thread.pool.size: "10"

    # Connection pool settings
    connection.pool.max.size: "100"
    connection.pool.timeout: "30000"
    connection.pool.min.idle: "5"
    connection.pool.max.lifetime: "300000"

    # Kafka metadata columns
    store.kafka.metadata: "true"

    # Auto-create tables with ReplacingMergeTree
    auto.create.tables: "true"
    auto.create.tables.replicated: "false"
    snowflake.id: "true"

    # Deduplication
    deduplication.policy: "off"

    # Consumer offset management
    consumer.auto.offset.reset: earliest

    # Error handling
    errors.tolerance: none
    errors.max.retries: "5"
    errors.log.enable: "true"
    errors.log.include.messages: "true"
    errors.deadletterqueue.topic.name: meeting-clickhouse-sink-dlq
    errors.deadletterqueue.topic.replication.factor: "2"
    errors.deadletterqueue.context.headers.enable: "true"

we modified develop branch to use kafka 4 and compiled and put it in kafka connect, e039b00

[Fluder-Paradyne]

also the db is mysq:8.0 in aws rds aurora

[Fluder-Paradyne]

please let me know if you need more details on our setup, also the connector in kafka connect https://repo1.maven.org/maven2/io/debezium/debezium-connector-mysql/3.2.0.Final/debezium-connector-mysql-3.2.0.Final-plugin.zip"

[aadant]

@subkanthi I think it is only working with the sink-connector lightweight for sub-second resolution ... so this issue is legit AFAIU. What do you reckon ? If so, we should use the same implementation as the lightweight to be consistent,