Skip to content

[BUG] automq-kafka 升级后消费数据的曲线少了 #2697

@codech

Description

@codech

AutoMQ Version

1.1.2

Operating System

TencentOS Server 3.1

Installation Method

binary

Hardware Configuration

8核cpu 64g内存

Other Relevant Software

No response

What Went Wrong?

1.1.2版本升级到1.4.1版本后报错,又回滚到1.1.2后发现消费端消费的数据曲线少了, 查看7天的可以明显的看出来. 并且还有报错

What Should Have Happened Instead?

[2025-01-17 14:35:59,488] WARN Failed to send 469 metrics for time 1737095756767 (kafka.autobalancer.metricsreporter.AutoBalancerMetricsReporter)
[2025-01-17 14:36:12,239] WARN Failed to send auto balancer metric (kafka.autobalancer.metricsreporter.AutoBalancerMetricsReporter)
org.apache.kafka.common.errors.OutOfOrderSequenceException: The broker received an out of order sequence number.
[2025-01-17 14:36:12,240] WARN Failed to send 531 metrics for time 1737095769488 (kafka.autobalancer.metricsreporter.AutoBalancerMetricsReporter)
[2025-01-17 14:40:16,914] INFO Closing Auto Balancer metrics reporter, id=3. (kafka.autobalancer.metricsreporter.AutoBalancerMetricsReporter)
[2025-01-17 14:40:16,914] INFO auto balancer metrics reporter exited. (kafka.autobalancer.metricsreporter.AutoBalancerMetricsReporter)
[2025-01-17 14:41:41,627] INFO Using default value of 172.21.64.42:9092 for autobalancer.reporter.bootstrap.servers (kafka.autobalancer.metricsreporter.AutoBalancerMetricsReporter)
[2025-01-17 14:41:41,680] INFO AutoBalancerMetricsReporter configuration finished (kafka.autobalancer.metricsreporter.AutoBalancerMetricsReporter)
[2025-01-17 14:41:41,682] INFO Starting auto balancer metrics reporter with reporting interval of 10000 ms. (kafka.autobalancer.metricsreporter.AutoBalancerMetricsReporter)
[2025-01-17 14:41:41,684] INFO AutoBalancerMetricsReporter init successful (kafka.autobalancer.metricsreporter.AutoBalancerMetricsReporter)
[2025-07-21 17:25:31,064] INFO Closing Auto Balancer metrics reporter, id=3. (kafka.autobalancer.metricsreporter.AutoBalancerMetricsReporter)
[2025-07-21 17:25:31,065] INFO auto balancer metric reporter is interrupted during flush due to shutdown request. (kafka.autobalancer.metricsreporter.AutoBalancerMetricsReporter)
[2025-07-21 17:25:31,065] INFO auto balancer metrics reporter exited. (kafka.autobalancer.metricsreporter.AutoBalancerMetricsReporter)
[2025-07-21 17:40:37,409] INFO Using default value of 172.21.64.42:9092 for autobalancer.reporter.bootstrap.servers (kafka.autobalancer.metricsreporter.AutoBalancerMetricsReporter)
[2025-07-21 17:40:37,450] INFO AutoBalancerMetricsReporter configuration finished (kafka.autobalancer.metricsreporter.AutoBalancerMetricsReporter)
[2025-07-21 17:40:37,452] INFO Starting auto balancer metrics reporter with reporting interval of 10000 ms. (kafka.autobalancer.metricsreporter.AutoBalancerMetricsReporter)
[2025-07-21 17:40:37,454] INFO AutoBalancerMetricsReporter init successful (kafka.autobalancer.metricsreporter.AutoBalancerMetricsReporter)
[2025-07-21 17:42:32,260] INFO Using default value of 172.21.64.42:9092 for autobalancer.reporter.bootstrap.servers (kafka.autobalancer.metricsreporter.AutoBalancerMetricsReporter)
[2025-07-21 17:42:32,304] INFO AutoBalancerMetricsReporter configuration finished (kafka.autobalancer.metricsreporter.AutoBalancerMetricsReporter)
[2025-07-21 17:42:32,305] INFO Starting auto balancer metrics reporter with reporting interval of 10000 ms. (kafka.autobalancer.metricsreporter.AutoBalancerMetricsReporter)
[2025-07-21 17:42:32,307] INFO AutoBalancerMetricsReporter init successful (kafka.autobalancer.metricsreporter.AutoBalancerMetricsReporter)
[2025-07-21 17:43:32,452] INFO Closing Auto Balancer metrics reporter, id=3. (kafka.autobalancer.metricsreporter.AutoBalancerMetricsReporter)
[2025-07-21 17:43:32,452] ERROR Got exception in auto balancer metrics reporter (kafka.autobalancer.metricsreporter.AutoBalancerMetricsReporter)
org.apache.kafka.common.errors.InterruptException: java.lang.InterruptedException
at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:1090)
at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:964)
at kafka.autobalancer.metricsreporter.AutoBalancerMetricsReporter.sendAutoBalancerMetric(AutoBalancerMetricsReporter.java:341)
at kafka.autobalancer.metricsreporter.AutoBalancerMetricsReporter.reportMetrics(AutoBalancerMetricsReporter.java:358)
at kafka.autobalancer.metricsreporter.AutoBalancerMetricsReporter.run(AutoBalancerMetricsReporter.java:294)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.lang.InterruptedException
at java.base/java.lang.Object.wait(Native Method)
at org.apache.kafka.common.utils.SystemTime.waitObject(SystemTime.java:55)
at org.apache.kafka.clients.producer.internals.ProducerMetadata.awaitUpdate(ProducerMetadata.java:120)
at org.apache.kafka.clients.producer.KafkaProducer.waitOnMetadata(KafkaProducer.java:1149)
at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:998)
... 5 more
[2025-07-21 17:43:32,452] INFO auto balancer metrics reporter exited. (kafka.autobalancer.metricsreporter.AutoBalancerMetricsReporter)
[2025-07-21 17:46:03,041] INFO Using default value of 172.21.64.42:9092 for autobalancer.reporter.bootstrap.servers (kafka.autobalancer.metricsreporter.AutoBalancerMetricsReporter)
[2025-07-21 17:46:03,106] INFO AutoBalancerMetricsReporter configuration finished (kafka.autobalancer.metricsreporter.AutoBalancerMetricsReporter)
[2025-07-21 17:46:03,108] INFO Starting auto balancer metrics reporter with reporting interval of 10000 ms. (kafka.autobalancer.metricsreporter.AutoBalancerMetricsReporter)
[2025-07-21 17:46:03,111] INFO AutoBalancerMetricsReporter init successful (kafka.autobalancer.metricsreporter.AutoBalancerMetricsReporter)
[2025-07-21 17:46:16,027] WARN Failed to send auto balancer metric (kafka.autobalancer.metricsreporter.AutoBalancerMetricsReporter)
org.apache.kafka.common.errors.NotLeaderOrFollowerException: For requests intended only for the leader, this error indicates that the broker is not the current leader. For requests intended for any replica, this error indicates that the broker is not a replica of the topic partition.
[2025-07-21 17:46:16,030] WARN Failed to send 2 metrics for time 1753091173110 (kafka.autobalancer.metricsreporter.AutoBalancerMetricsReporter)

Steps to Reproduce

升级操作

Additional Information

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions