[opt](cloud) Prioritize scheduling the most recently active tablets in cloud #59539
+1,504
−15
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
deardeng
requested review from
dataroaring,
gavinchou and
w41ter
as code owners
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
deardeng
force-pushed
the
active-tablet
branch
from
91eef60 to
2f8bdee
Compare
deardeng
force-pushed
the
active-tablet
branch
from
ac383bb to
a71331b
Compare
Contributor
Author
|
run buildall |
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 31449 ms |
TPC-DS: Total hot run time: 176500 ms |
gavinchou
reviewed
Jan 9, 2026
|
|
||
| // Bucket size in milliseconds (default: 1 minute) | ||
| // Smaller bucket = more accurate but more memory | ||
| private final long bucketSizeMs; |
gavinchou
reviewed
Jan 9, 2026
| .thenComparing(Comparator.comparingLong((AccessStatsResult r) -> r.lastAccessTime).reversed()); | ||
|
|
||
| // Time window in milliseconds (default: 1 hour) | ||
| private final long timeWindowMs; |
Contributor
There was a problem hiding this comment.
what is this time window for?
gavinchou
reviewed
Jan 10, 2026
| private ThreadPoolExecutor asyncExecutor; | ||
|
|
||
| // Default time window: 1 hour | ||
| private static final long DEFAULT_TIME_WINDOW_SECOND = 3600L; |
gavinchou
reviewed
Jan 10, 2026
| * Get shard index for a given ID | ||
| */ | ||
| private int getShardIndex(long id) { | ||
| return (int) (id & (SHARD_SIZE - 1)); |
Contributor
There was a problem hiding this comment.
better use some hash function, e.g. crc/md5, on id first and then take modulo
gavinchou
reviewed
Jan 10, 2026
| */ | ||
| public long getTabletIdByReplicaId(long replicaId) { | ||
| // Use timeout to prevent deadlock, default timeout: 2 seconds | ||
| long timeoutMs = 2000L; |
Contributor
There was a problem hiding this comment.
what is the magic 2000? why not 500 or 1000?
gavinchou
reviewed
Jan 10, 2026
| * Result for top N query | ||
| */ | ||
| public static class AccessStatsResult { | ||
| public final long id; |
gavinchou
reviewed
Jan 10, 2026
| long currentTime = System.currentTimeMillis(); | ||
| // Record tablet access | ||
| // Get real tabletId from TabletInvertedIndex instead of calculating it | ||
| long tabletId = Env.getCurrentInvertedIndex().getTabletIdByReplicaId(replicaId); |
Contributor
There was a problem hiding this comment.
is there any performance issue with inverted idex accessing?
gavinchou
reviewed
Jan 10, 2026
| } catch (Exception e) { | ||
| // If executor is shutdown or queue is full, silently ignore | ||
| // Statistics can tolerate some loss | ||
| if (LOG.isDebugEnabled()) { |
Contributor
There was a problem hiding this comment.
log info here, exception here is abnormal
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Starting from the getBackendId of the replica, implement a sliding window approach (in TabletAccessStats.java) to count read and write operations over the last hour. The accuracy of this count isn't critical, as the aim is to indicate relative activity levels.
Display these statistics in the SHOW TABLETS FROM TABLE and SHOW TABLET {tabletId} commands.
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)