Merge LastQueryScanNode of same device #15735

shuwenwei · 2025-06-17T01:55:47Z

Description

Merge LastQueryScanNode of same device
For a scenario with 31185 non-aligned devices and 1 measurement points per device.
When the last cache is hit, the optimization effect of executing the following SQL statement is:

select last value from root.reactor.**

Before this optimization:

img_v3_02np_75b07e13-a474-47d6-b5b1-dcf837e07f9g

After this optimization:

For a scenario with 2 * 2 * 4 non-aligned devices and 80,000 measurement points per device.
When the last cache is hit, the optimization effect of executing the following SQL statement is:

select last * from root.db.aaaaaaa0.bbbbbbb0.**

Before this optimization:

After this optimization:

codecov · 2025-06-17T07:22:58Z

Codecov Report

Attention: Patch coverage is 59.71223% with 168 lines in your changes missing coverage. Please review.

Project coverage is 39.11%. Comparing base (b8fd089) to head (352191f).
Report is 21 commits behind head on master.

Files with missing lines	Patch %	Lines
...ueryengine/plan/planner/OperatorTreeGenerator.java	0.00%	45 Missing ⚠️
...an/planner/plan/node/source/LastQueryScanNode.java	52.94%	32 Missing ⚠️
.../planner/plan/node/process/last/LastQueryNode.java	55.22%	30 Missing ⚠️
.../process/last/AbstractUpdateLastCacheOperator.java	13.33%	13 Missing ⚠️
...db/db/queryengine/plan/analyze/AnalyzeVisitor.java	70.58%	10 Missing ⚠️
...b/queryengine/plan/planner/LogicalPlanBuilder.java	74.35%	10 Missing ⚠️
...gine/plan/planner/distribution/SourceRewriter.java	88.88%	7 Missing ⚠️
...ngine/execution/fragment/DataNodeQueryContext.java	16.66%	5 Missing ⚠️
...er/distribution/SimpleFragmentParallelPlanner.java	16.66%	5 Missing ⚠️
...e/iotdb/db/queryengine/common/MPPQueryContext.java	81.81%	2 Missing ⚠️
... and 6 more

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #15735      +/-   ##
============================================
+ Coverage     39.09%   39.11%   +0.01%     
  Complexity      198      198              
============================================
  Files          4839     4839              
  Lines        314856   315188     +332     
  Branches      39530    39567      +37     
============================================
+ Hits         123108   123300     +192     
- Misses       191748   191888     +140

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

JackieTien97 · 2025-07-01T03:41:12Z

iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/partition/DataPartition.java

+  public List<TRegionReplicaSet> getDataRegionReplicaSetWithTimeFilter(
+      String db, TSeriesPartitionSlot tSeriesPartitionSlot, Filter timeFilter) {
+    Map<TTimePartitionSlot, List<TRegionReplicaSet>> regionReplicaSetMap =
+        dataPartitionMap
+            .getOrDefault(db, Collections.emptyMap())
+            .getOrDefault(tSeriesPartitionSlot, Collections.emptyMap());
+    if (regionReplicaSetMap.isEmpty()) {
+      return Collections.singletonList(NOT_ASSIGNED);
+    }
+    List<TRegionReplicaSet> replicaSets = new ArrayList<>();
+    Set<TRegionReplicaSet> uniqueValues = new HashSet<>();
+    for (Entry<TTimePartitionSlot, List<TRegionReplicaSet>> entry :
+        regionReplicaSetMap.entrySet()) {
+      if (!TimePartitionUtils.satisfyPartitionStartTime(timeFilter, entry.getKey().startTime)) {
+        continue;
+      }
+      for (TRegionReplicaSet tRegionReplicaSet : entry.getValue()) {
+        if (uniqueValues.add(tRegionReplicaSet)) {
+          replicaSets.add(tRegionReplicaSet);
+        }
+      }
+    }
+    return replicaSets;
+  }
+


JackieTien97 · 2025-07-01T03:44:46Z

.../src/test/java/org/apache/iotdb/db/queryengine/execution/operator/LastQueryOperatorTest.java

          new LastQueryOperator(
              driverContext.getOperatorContexts().get(4),
-              ImmutableList.of(updateLastCacheOperator1, updateLastCacheOperator2),
+              Arrays.asList(updateLastCacheOperator1, updateLastCacheOperator2),


why we need to change it into a mutable list?

JackieTien97 · 2025-07-01T03:47:09Z

.../java/org/apache/iotdb/db/queryengine/execution/operator/process/last/LastQueryOperator.java

          return null;
        } else if (!tsBlock.isEmpty()) {
          LastQueryUtil.appendLastValue(tsBlockBuilder, tsBlock);
+          continue;


child.next can only be called once in parent's next call

return null;

JackieTien97 · 2025-07-01T03:58:30Z

...a/org/apache/iotdb/db/queryengine/execution/operator/process/last/LastQuerySortOperator.java

        previousTsBlockIndex = 0;
-        if (previousTsBlock == null) {
-          return true;
+        if (previousTsBlock == null || children.get(currentIndex).hasNextWithTimer()) {


we should only call hasNextWithTimer once

just return

...core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/analyze/AnalyzeVisitor.java

.../java/org/apache/iotdb/db/queryengine/plan/planner/plan/node/process/last/LastQueryNode.java

JackieTien97 · 2025-07-01T06:02:06Z

.../java/org/apache/iotdb/db/queryengine/plan/planner/plan/node/process/last/LastQueryNode.java

  // if children contains LastTransformNode, this variable is only used in distribute plan
  private boolean containsLastTransformNode;

+  private Map<IMeasurementSchema, Integer> measurementSchema2IdxMap;


add some comments about this field to describe its life cycle, like when will this field be set to null

.../java/org/apache/iotdb/db/queryengine/plan/planner/plan/node/process/last/LastQueryNode.java

JackieTien97 · 2025-07-01T06:12:57Z

.../java/org/apache/iotdb/db/queryengine/plan/planner/plan/node/process/last/LastQueryNode.java

  @Override
-  public List<PlanNode> getChildren() {
-    return children;
+  public void serialize(DataOutputStream stream) throws IOException {
+    super.serialize(stream);
  }

  @Override
-  public void addChild(PlanNode child) {
-    children.add(child);
+  public void serialize(ByteBuffer byteBuffer) {
+    super.serialize(byteBuffer);
  }


don't need to override it?

...in/java/org/apache/iotdb/db/queryengine/plan/planner/plan/node/source/LastQueryScanNode.java

…ine/plan/analyze/AnalyzeVisitor.java Co-authored-by: Jackie Tien <[email protected]>

…ine/plan/planner/plan/node/process/last/LastQueryNode.java Co-authored-by: Jackie Tien <[email protected]>

…ine/plan/planner/plan/node/source/LastQueryScanNode.java Co-authored-by: Jackie Tien <[email protected]>

…ine/plan/planner/plan/node/process/last/LastQueryNode.java Co-authored-by: Jackie Tien <[email protected]>

sonarqubecloud · 2025-07-01T11:22:17Z

Quality Gate passed

Issues
22 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.3% Duplication on New Code

See analysis details on SonarQube Cloud

shuwenwei added 5 commits June 16, 2025 19:04

add DeviceLastQueryScanNode

2955c26

Merge branch 'iotdb_master' into mergeLastQueryScanNodeOfSameDevice

335e174

fix deserialize

115c41f

remove deviceAlignedSet

48e3b19

fix unsupported operation in ut

ce2593b

shuwenwei added 12 commits June 18, 2025 11:05

fix last query sort operator

0f3201a

Merge branch 'iotdb_master' into mergeLastQueryScanNodeOfSameDevice

732b459

remove debug code

c0ed35b

modify LastQueryNode

075596a

update other stage of last query

46f865b

Merge branch 'iotdb_master' into mergeLastQueryScanNodeOfSameDevice

b4365e3

rename

b0484ef

optimize distribution plan and dispatch

dab659d

fix test and optimize SourceRewriter

6fb670c

fix bug for view

6e4e5d0

fix it

b3c00f6

remove useless code

ecfbe2f

JackieTien97 requested changes Jul 1, 2025

View reviewed changes

...in/java/org/apache/iotdb/db/queryengine/plan/planner/plan/node/source/LastQueryScanNode.java Outdated Show resolved Hide resolved

shuwenwei and others added 8 commits July 1, 2025 16:29

Update iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryeng…

0c59535

…ine/plan/analyze/AnalyzeVisitor.java Co-authored-by: Jackie Tien <[email protected]>

Update iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryeng…

c71384d

…ine/plan/planner/plan/node/process/last/LastQueryNode.java Co-authored-by: Jackie Tien <[email protected]>

Update iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryeng…

5e4330d

…ine/plan/planner/plan/node/process/last/LastQueryNode.java Co-authored-by: Jackie Tien <[email protected]>

Update iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryeng…

7530aca

…ine/plan/planner/plan/node/process/last/LastQueryNode.java Co-authored-by: Jackie Tien <[email protected]>

Update iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryeng…

9af3248

…ine/plan/planner/plan/node/source/LastQueryScanNode.java Co-authored-by: Jackie Tien <[email protected]>

Update iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryeng…

2755e16

…ine/plan/planner/plan/node/process/last/LastQueryNode.java Co-authored-by: Jackie Tien <[email protected]>

fix review

9df1a73

fix ut

81d3854

add comment

352191f

JackieTien97 approved these changes Jul 2, 2025

View reviewed changes

JackieTien97 merged commit d86b86e into master Jul 2, 2025
60 of 66 checks passed

JackieTien97 deleted the mergeLastQueryScanNodeOfSameDevice branch July 2, 2025 10:37

shuwenwei mentioned this pull request Jul 3, 2025

[To dev/1.3] Merge last query scan node of same device #15863

Merged

Merge LastQueryScanNode of same device #15735

Merge LastQueryScanNode of same device #15735

Uh oh!

Conversation

shuwenwei commented Jun 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Uh oh!

codecov bot commented Jun 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

JackieTien97 Jul 1, 2025

Choose a reason for hiding this comment

Uh oh!

JackieTien97 Jul 1, 2025

Choose a reason for hiding this comment

Uh oh!

JackieTien97 Jul 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JackieTien97 Jul 1, 2025

Choose a reason for hiding this comment

Uh oh!

JackieTien97 Jul 1, 2025

Choose a reason for hiding this comment

Uh oh!

JackieTien97 Jul 1, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

JackieTien97 Jul 1, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

JackieTien97 Jul 1, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sonarqubecloud bot commented Jul 1, 2025

Quality Gate passed

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

shuwenwei commented Jun 17, 2025 •

edited

Loading

codecov bot commented Jun 17, 2025 •

edited

Loading

JackieTien97 Jul 1, 2025 •

edited

Loading