vivekchavan14/fix autobalancer metrics reporter outofordersequence #2700
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Fix IllegalArgumentException: Illegal base64 character 20 in S3StreamKafkaMetricsManager
- Replace single newline removal with comprehensive whitespace cleanup using replaceAll("\s", "")
- Add graceful error handling for both Base64 and certificate parsing failures
- Add comprehensive unit tests covering various whitespace scenarios and edge cases
- Improve logging with specific error messages for failed certificate parsing
Fixes AutoMQ#2615
…trics reporter robustness - Enable producer idempotence to prevent OutOfOrderSequenceException during retries - Add proper timeout configurations (delivery_timeout_ms=120000, request_timeout_ms=30000) - Improve error handling with specific exception type logging - Implement graceful shutdown with producer flush and thread join - Add shutdown guards to prevent metrics sending during shutdown - Enhance tests for configuration validation and shutdown scenarios Fixes AutoMQ#2697: Missing consumption data curves and OutOfOrderSequenceException errors This resolves the issues reported after upgrading from v1.1.2 to v1.4.1 where: - OutOfOrderSequenceException caused metric transmission failures - Missing idempotence configuration led to sequence number conflicts during retries - Poor shutdown handling caused InterruptExceptions - Generic error handling made troubleshooting difficult The fix maintains backward compatibility while significantly improving reliability.
vivekchavan14
requested review from
superhx,
Gezi-lzq,
1sonofqiu and
woshigaopp
as code owners
vivekchavan14
changed the title
|
Welcome to use AI to fix issues, but the expected fixes need to be verified and confirmed by the developers themselves. @vivekchavan14 |
|
Yup, I’ve reviewed the response, it's not entirely AI-generated. I used AI to assist with the analysis, but all suggested fixes and insights have been manually verified and cross-checked before posting. Let me know if any part needs further clarification or adjustment. |
yx9o
reviewed
Aug 22, 2025
| The root cause was that the AutoBalancer metrics reporter was configured with: | ||
| - `retries = 5` | ||
| - `acks = all` | ||
| - **BUT missing `enable.idempotence = true`** |
There was a problem hiding this comment.
The default here is true, and there is no conflicting configuration. Idempotence should be effective and no additional modification is required.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
Fixes #2697 - AutoBalancer metrics reporter experiencing OutOfOrderSequenceException and missing consumption data curves after upgrading from v1.1.2 to v1.4.1.
Root Cause
The producer was configured with retries and acks=all but missing enable.idempotence=true, causing sequence number conflicts during retries.
Changes Made