HIP-1081 Prioritize block nodes by latency #11607

xin-hedera · 2025-07-18T14:25:50Z

Description:

This PR implements latency based block node scheduling

Add three block node schedulers: LATENCY, PRIORITY, and PRIORITY_THEN_LATENCY
Add LatencyService to measure block node streaming latency in background for latency aware schedulers

Related issue(s):

Fixes #11271
Fixes #11546

Notes for reviewer:

Checklist

Documented (Code comments, README, etc.)
Tested (unit, integration, etc.)

Signed-off-by: Xin Li <[email protected]>

lfdt-bot · 2025-07-18T14:26:08Z

✅ Snyk checks have passed. No issues have been found so far.

Status	Scanner	Critical	High	Medium	Low	Total (0)
✅	Open Source Security	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

Signed-off-by: Xin Li <[email protected]>

codacy-production · 2025-07-18T16:09:40Z

Coverage summary from Codacy

See diff coverage on Codacy

Coverage variation	Diff coverage
❌ -39.07% (target: -1.00%)	✅ 97.62%

Coverage variation details

	Coverable lines	Covered lines	Coverage
Common ancestor commit (`64ab944`)	27275	25606	93.88%
Head commit (`48110dc`)	52936 (+25661)	29015 (+3409)	54.81% (-39.07%)

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details

	Coverable lines	Covered lines	Diff coverage
Pull request (#11607)	252	246	97.62%

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

See your quality gate settings Change summary preferences

Signed-off-by: Xin Li <[email protected]>

docs/configuration.md

importer/src/main/java/org/hiero/mirror/importer/downloader/block/BlockNode.java

importer/src/main/java/org/hiero/mirror/importer/downloader/block/BlockNodeSubscriber.java

…ency Signed-off-by: Xin Li <[email protected]>

Signed-off-by: Xin Li <[email protected]>

…ency Signed-off-by: Xin Li <[email protected]>

Signed-off-by: Xin Li <[email protected]>

…tize-block-nodes-by-latency Signed-off-by: Xin Li <[email protected]>

Signed-off-by: Xin Li <[email protected]>

…ency Signed-off-by: Xin Li <[email protected]>

Signed-off-by: Xin Li <[email protected]>

…ency Signed-off-by: Xin Li <[email protected]>

Signed-off-by: Xin Li <[email protected]>

…ency Signed-off-by: Xin Li <[email protected]>

importer/src/main/java/org/hiero/mirror/importer/downloader/block/BlockNodeSubscriber.java

importer/src/main/java/org/hiero/mirror/importer/downloader/block/Latency.java

…ency Signed-off-by: Xin Li <[email protected]>

Signed-off-by: Xin Li <[email protected]>

steven-sheehy · 2026-01-21T23:36:46Z

docs/configuration.md

+| `hiero.mirror.importer.block.scheduler.maxPostProcessingLatency`                | 1s                                                   | The maximum allowed post-processing delay to calculate and record block node streaming latency.                                                                                                                                                                    |
+| `hiero.mirror.importer.block.scheduler.minRescheduleInterval`                   | 10s                                                  | The mininum block node reschedule interval.                                                                                                                                                                                                                        |
+| `hiero.mirror.importer.block.scheduler.rescheduleLatencyThreshold`              | 50ms                                                 | The threshold to meet for lower latency block nodes to trigger a reschedule.                                                                                                                                                                                       |
+| `hiero.mirror.importer.block.scheduler.type`                                    | PRIORITY                                             | The scheduler type. Can be `LATENCY`, `PRIORITY`, or `PRIORITY_THEN_LATENCY`.                                                                                                                                                                                      |


Shouldn't we default to PRIORITY_THEN_LATENCY? Otherwise all this latency work will never be used in practice.

steven-sheehy · 2026-01-21T23:38:44Z

docs/configuration.md

 | `hiero.mirror.importer.block.nodes[].port`                                      | 40840                                                | The port of the block node server.                                                                                                                                                                                                                                 |
 | `hiero.mirror.importer.block.nodes[].priority`                                  | 0                                                    | The priority of the block node server. A lower value indicates higher priority, and 0 is the highest priority.                                                                                                                                                     |
 | `hiero.mirror.importer.block.persistBytes`                                      | false                                                | Whether to persist the block stream file bytes to the database.                                                                                                                                                                                                    |
+| `hiero.mirror.importer.block.scheduler.latencyService.backlog`                  | 1                                                    | The backlog size of pending latency measuring tasks. Note a max number of backlog plus 1 tasks can be scheduled when the service is idle.                                                                                                                          |


Prefer simpler latency for property names instead of implementation names.

Suggested change

| `hiero.mirror.importer.block.scheduler.latencyService.backlog` | 1 | The backlog size of pending latency measuring tasks. Note a max number of backlog plus 1 tasks can be scheduled when the service is idle. |

| `hiero.mirror.importer.block.scheduler.latency.backlog` | 1 | The backlog size of pending latency measuring tasks. Note a max number of backlog plus 1 tasks can be scheduled when the service is idle. |

steven-sheehy · 2026-01-22T01:05:07Z

importer/src/main/java/org/hiero/mirror/importer/downloader/block/Latency.java

+final class Latency {
+
+    private static final int HISTORY_SIZE = 5;
+
+    @Getter
+    private long average = Long.MIN_VALUE;
+
+    private int count = 0;
+    private final long[] history = new long[HISTORY_SIZE];


This class seems like overkill and may not smooth the averages as we like. Why can't we drop the class and use a simple exponential moving average in the calling class that's stored as a double or AtomicDouble?

steven-sheehy · 2026-01-22T01:12:02Z

importer/src/main/java/org/hiero/mirror/importer/downloader/block/SchedulerProperties.java

+
+@Data
+@Validated
+public final class SchedulerProperties {


Prefer this as a separate @ConfigurationProperties instead of nested to reduce coupling. Also move to scheduler package.

steven-sheehy · 2026-01-30T01:50:52Z

importer/src/main/java/org/hiero/mirror/importer/downloader/block/BlockNode.java

+                    case END_OF_BLOCK -> {
+                        running = !assembler.onEndOfBlock(response.getEndOfBlock());
+                        if (!running) {
+                            log.info("Cancel the subscription to try rescheduling");


This log sounds like a suggestion to the operator to take some action. Do you mean to say Cancelling the subscription... to indicate the code is doing the action?

Suggested change

log.info("Cancel the subscription to try rescheduling");

log.info("Cancel the subscription to try rescheduling");

steven-sheehy · 2026-02-03T05:05:30Z

...java/org/hiero/mirror/importer/downloader/block/scheduler/AbstractLatencyAwareScheduler.java

+    protected final AtomicReference<@Nullable BlockNode> current = new AtomicReference<>();
+    protected final AtomicLong lastScheduledTime = new AtomicLong(0);
+
+    protected long lastPostProcessingLatency;


Can be private. Should also be atomic or volatile to be safe.

steven-sheehy · 2026-02-03T05:11:29Z

importer/src/main/java/org/hiero/mirror/importer/downloader/block/scheduler/LatencyService.java

+    @Scheduled(fixedDelayString = "#{@blockProperties.getScheduler().getLatencyService().getFrequency().toMillis()}")
+    public void schedule() {


Is it possible this takes longer than frequency and multiple invocations occur simultaneously? If so, might need synchronized.

steven-sheehy · 2026-02-03T05:13:11Z

...java/org/hiero/mirror/importer/downloader/block/scheduler/AbstractLatencyAwareScheduler.java

+            latencyService.cancelAll();
+            current.set(super.getNode(blockNumber));
+            candidates.clear();
+            candidates.addAll(getCandidates());
+            latencyService.setNodes(candidates);


latencyService.cancelAll() called twice: once directly and other via setNodes.

steven-sheehy · 2026-02-03T05:16:30Z

importer/src/main/java/org/hiero/mirror/importer/downloader/block/scheduler/LatencyService.java

+        cancelAll();
+
+        long bornGeneration = generation.incrementAndGet();
+        nodes.forEach(blockNode -> tasks.add(new Task(bornGeneration, blockNode)));


Is it possible to optimize to skip currently connected node since we can measure that directly elsewhere?

steven-sheehy · 2026-02-03T05:19:21Z

importer/src/main/java/org/hiero/mirror/importer/downloader/block/scheduler/LatencyService.java

+            log.info("Measuring {}'s latency by streaming block {}", node, nextBlockNumber);
+            final var timeout =
+                    blockProperties.getScheduler().getLatencyService().getTimeout();
+            node.streamBlocks(nextBlockNumber, nextBlockNumber, this::measureLatency, timeout);


The next block might not be available. Should we stream a couple previous blocks that we know are present? This would omit time waiting for next block and spread the latency across a few blocks.

implement latency based block node scheduling

c481a9b

Signed-off-by: Xin Li <[email protected]>

xin-hedera self-assigned this Jul 18, 2025

xin-hedera linked an issue Jul 18, 2025 that may be closed by this pull request

HIP-1081 Prioritize block nodes by latency #11271

Open

xin-hedera added the enhancement Type: New feature label Jul 18, 2025

xin-hedera added this to the 0.135.0 milestone Jul 18, 2025

xin-hedera added 2 commits July 18, 2025 09:58

use Long.MIN_VALUE as the initial average latency

36d7550

Signed-off-by: Xin Li <[email protected]>

reorder the priority group everytime a node's latency is updated

70e5fdc

Signed-off-by: Xin Li <[email protected]>

remove / replace NotNull

e1f6538

Signed-off-by: Xin Li <[email protected]>

steven-sheehy modified the milestones: 0.135.0, 0.136.0 Jul 23, 2025

steven-sheehy added the importer Area: Importer label Jul 25, 2025

steven-sheehy reviewed Jul 25, 2025

View reviewed changes

Merge branch 'main' into 11271-hip-1081-prioritize-block-nodes-by-lat…

a20cfc0

…ency Signed-off-by: Xin Li <[email protected]>

steven-sheehy removed this from the 0.136.0 milestone Aug 6, 2025

xin-hedera added 8 commits August 11, 2025 16:07

Merge branch 'main' into 11271-hip-1081-prioritize-block-nodes-by-lat…

0a6414b

…ency Signed-off-by: Xin Li <[email protected]>

add schedulers, fix existing test cases

00861b0

Signed-off-by: Xin Li <[email protected]>

Merge branch 'main' into 11271-hip-1081-prioritize-block-nodes-by-lat…

b6be714

…ency Signed-off-by: Xin Li <[email protected]>

latency scheduler and test cases

c9fc66f

Signed-off-by: Xin Li <[email protected]>

Merge remote-tracking branch 'origin/main' into 11271-hip-1081-priori…

76d50f6

…tize-block-nodes-by-latency Signed-off-by: Xin Li <[email protected]>

fix failing tests

4c3738a

Signed-off-by: Xin Li <[email protected]>

switch to inprocess channel and try to stablize the tests

c565580

Signed-off-by: Xin Li <[email protected]>

Merge branch 'main' into 11271-hip-1081-prioritize-block-nodes-by-lat…

e65fe72

…ency Signed-off-by: Xin Li <[email protected]>

xin-hedera added this to the 0.147.0 milestone Jan 14, 2026

xin-hedera added 3 commits January 14, 2026 09:12

minor

3ee0b28

Signed-off-by: Xin Li <[email protected]>

Merge branch 'main' into 11271-hip-1081-prioritize-block-nodes-by-lat…

ec9ee5a

…ency Signed-off-by: Xin Li <[email protected]>

fix SpEL

1594416

Signed-off-by: Xin Li <[email protected]>

xin-hedera marked this pull request as ready for review January 14, 2026 16:40

xin-hedera requested a review from a team as a code owner January 14, 2026 16:40

xin-hedera requested a review from ashumahajan January 14, 2026 16:40

Merge branch 'main' into 11271-hip-1081-prioritize-block-nodes-by-lat…

8a45451

…ency Signed-off-by: Xin Li <[email protected]>

ashumahajan reviewed Jan 20, 2026

View reviewed changes

importer/src/main/java/org/hiero/mirror/importer/downloader/block/BlockNodeSubscriber.java Show resolved Hide resolved

importer/src/main/java/org/hiero/mirror/importer/downloader/block/Latency.java Show resolved Hide resolved

xin-hedera modified the milestones: 0.147.0, 0.148.0 Jan 22, 2026

xin-hedera added 2 commits January 23, 2026 11:48

Merge branch 'main' into 11271-hip-1081-prioritize-block-nodes-by-lat…

4dfccc0

…ency Signed-off-by: Xin Li <[email protected]>

add integration test coverage for diffrent status & streaming ports

ddca603

Signed-off-by: Xin Li <[email protected]>

xin-hedera force-pushed the 11271-hip-1081-prioritize-block-nodes-by-latency branch from 7bbef9c to ddca603 Compare January 23, 2026 19:22

spotless apply

48110dc

Signed-off-by: Xin Li <[email protected]>

jnels124 approved these changes Jan 28, 2026

View reviewed changes

steven-sheehy requested changes Feb 3, 2026

View reviewed changes

	\| `hiero.mirror.importer.block.scheduler.latencyService.backlog` \| 1 \| The backlog size of pending latency measuring tasks. Note a max number of backlog plus 1 tasks can be scheduled when the service is idle. \|
	\| `hiero.mirror.importer.block.scheduler.latency.backlog` \| 1 \| The backlog size of pending latency measuring tasks. Note a max number of backlog plus 1 tasks can be scheduled when the service is idle. \|

	log.info("Cancel the subscription to try rescheduling");
	log.info("Cancel the subscription to try rescheduling");

		@Scheduled(fixedDelayString = "#{@blockProperties.getScheduler().getLatencyService().getFrequency().toMillis()}")
		public void schedule() {

HIP-1081 Prioritize block nodes by latency #11607

Are you sure you want to change the base?

HIP-1081 Prioritize block nodes by latency #11607

Uh oh!

Conversation

xin-hedera commented Jul 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lfdt-bot commented Jul 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Snyk checks have passed. No issues have been found so far.

Uh oh!

codacy-production bot commented Jul 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Coverage summary from Codacy

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

xin-hedera commented Jul 18, 2025 •

edited

Loading

lfdt-bot commented Jul 18, 2025 •

edited

Loading

codacy-production bot commented Jul 18, 2025 •

edited

Loading