vivekchavan14/fix autobalancer metrics reporter outofordersequence #2700
Conversation
- Fix IllegalArgumentException: Illegal base64 character 20 in S3StreamKafkaMetricsManager
- Replace single newline removal with comprehensive whitespace cleanup using replaceAll("\s", "")
- Add graceful error handling for both Base64 and certificate parsing failures
- Add comprehensive unit tests covering various whitespace scenarios and edge cases
- Improve logging with specific error messages for failed certificate parsing
Fixes AutoMQ#2615
…trics reporter robustness - Enable producer idempotence to prevent OutOfOrderSequenceException during retries - Add proper timeout configurations (delivery_timeout_ms=120000, request_timeout_ms=30000) - Improve error handling with specific exception type logging - Implement graceful shutdown with producer flush and thread join - Add shutdown guards to prevent metrics sending during shutdown - Enhance tests for configuration validation and shutdown scenarios Fixes AutoMQ#2697: Missing consumption data curves and OutOfOrderSequenceException errors This resolves the issues reported after upgrading from v1.1.2 to v1.4.1 where: - OutOfOrderSequenceException caused metric transmission failures - Missing idempotence configuration led to sequence number conflicts during retries - Poor shutdown handling caused InterruptExceptions - Generic error handling made troubleshooting difficult The fix maintains backward compatibility while significantly improving reliability.
vivekchavan14
requested review from
1sonofqiu,
Gezi-lzq,
superhx and
woshigaopp
as code owners
vivekchavan14
changed the title
|
Welcome to use AI to fix issues, but the expected fixes need to be verified and confirmed by the developers themselves. @vivekchavan14 |
|
Yup, I’ve reviewed the response, it's not entirely AI-generated. I used AI to assist with the analysis, but all suggested fixes and insights have been manually verified and cross-checked before posting. Let me know if any part needs further clarification or adjustment. |
| The root cause was that the AutoBalancer metrics reporter was configured with: | ||
| - `retries = 5` | ||
| - `acks = all` | ||
| - **BUT missing `enable.idempotence = true`** |
There was a problem hiding this comment.
The default here is true, and there is no conflicting configuration. Idempotence should be effective and no additional modification is required.
|
This PR is being marked as stale since it has not had any activity in 90 days. If you would like to keep this PR alive, please ask a committer for review. If the PR has merge conflicts, please update it with the latest from trunk (or appropriate release branch) If this PR is no longer valid or desired, please feel free to close it. If no activity occurs in the next 30 days, it will be automatically closed. |
3 participants
Problem
Fixes #2697 - AutoBalancer metrics reporter experiencing OutOfOrderSequenceException and missing consumption data curves after upgrading from v1.1.2 to v1.4.1.
Root Cause
The producer was configured with retries and acks=all but missing enable.idempotence=true, causing sequence number conflicts during retries.
Changes Made