SNOW-1865904 fix query gen when nested cte node is partitioned#2816
Conversation
sfc-gh-helmeleegy
left a comment
There was a problem hiding this comment.
LGTM, just a couple of nits. Thanks!
Co-authored-by: Hazem Elmeleegy <hazem.elmeleegy@snowflake.com>
| nodes_to_reset = list(parents) | ||
| while nodes_to_reset: | ||
| node = nodes_to_reset.pop() | ||
| if node in updated_nodes: |
There was a problem hiding this comment.
can you give some more detailed description about what is the problem if we skip the node update here ? the cte optimizaiton is doing a level to level update, so it can skip the node update if it is already updated, but for large query breakdown, i recall you were doing a dfs, right?
There was a problem hiding this comment.
consider a tree below
node1
|
node2
/ \
node3 node4
suppose we start by re-resolving node3 -> node2 -> node1. In this process, node2 and node1 are marked in updated_nodes. Now, when we go updating the ancestors of node4, re-resolution of node2 and node1 would be skipped. This is not ideal if node4 update can also trigger a re-update of node2 and node1.
For example, this could be problematic is when node3 and node4 before the update had referenced_cte map. After the first update node3 -> node2 -> node1, node2 will be resolved with an older version of node4. If after a re-resolved node4 there are no referenced_ctes, then node2 will not be updated with this information.
There was a problem hiding this comment.
so it is because you are doing a dfs, if you are doing updated in the order of node3, node4 then node2, last node1 you shouldn't run into problems.
| error_type=type(e).__name__, | ||
| error_message=str(e), | ||
| ) | ||
| pass |
There was a problem hiding this comment.
do we have a bug before?
There was a problem hiding this comment.
no. I just noticed an unnecessary pass so I cleaned it up.
…-cte-node-is-partitioned
Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR.
Fixes SNOW-1865904
Fill out the following pre-review checklist:
Please describe how your code solves the related issue.
Update query generation for with query block where cte query is updated if the child plan is re-resolved during optimization process.