Commit ee41857
### What changes were proposed in this pull request?
Clean revert of d65234b.
Will later handle for cases of sourceSide child nodes without `numOutputRows`, and will re-target the new implementation to later Spark release.
### Why are the changes needed?
The current implementation may grab the incorrect `numOutputRows` metric if there is an intermediary node (such as custom Spark operator) which does not support the metric.
This is because we target the first sourceSide child node with `numOutputRows`. If a SparkExtension node does not contain this metric but transforms the source table, then we could progress all the way to the source table and grab the incorrect metric.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Existing CI, as this is a revert
### Was this patch authored or co-authored using generative AI tooling?
No
Closes #53293 from asl3/numsourcerowsrevert.
Authored-by: Amanda Liu <[email protected]>
Signed-off-by: Wenchen Fan <[email protected]>
1 parent 9969548 commit ee41857
File tree
4 files changed
+227
-346
lines changed- sql
- catalyst/src/main
- java/org/apache/spark/sql/connector/write
- scala/org/apache/spark/sql/connector/write
- core/src
- main/scala/org/apache/spark/sql/execution/datasources/v2
- test/scala/org/apache/spark/sql/connector
4 files changed
+227
-346
lines changedLines changed: 0 additions & 5 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
27 | 27 | | |
28 | 28 | | |
29 | 29 | | |
30 | | - | |
31 | | - | |
32 | | - | |
33 | | - | |
34 | | - | |
35 | 30 | | |
36 | 31 | | |
37 | 32 | | |
| |||
Lines changed: 0 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
21 | 21 | | |
22 | 22 | | |
23 | 23 | | |
24 | | - | |
25 | 24 | | |
26 | 25 | | |
27 | 26 | | |
| |||
Lines changed: 1 addition & 38 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
31 | 31 | | |
32 | 32 | | |
33 | 33 | | |
34 | | - | |
| 34 | + | |
35 | 35 | | |
36 | 36 | | |
37 | 37 | | |
38 | | - | |
39 | 38 | | |
40 | 39 | | |
41 | 40 | | |
| |||
493 | 492 | | |
494 | 493 | | |
495 | 494 | | |
496 | | - | |
497 | 495 | | |
498 | | - | |
499 | 496 | | |
500 | 497 | | |
501 | 498 | | |
| |||
507 | 504 | | |
508 | 505 | | |
509 | 506 | | |
510 | | - | |
511 | | - | |
512 | | - | |
513 | | - | |
514 | | - | |
515 | | - | |
516 | | - | |
517 | | - | |
518 | | - | |
519 | | - | |
520 | | - | |
521 | | - | |
522 | | - | |
523 | | - | |
524 | | - | |
525 | | - | |
526 | | - | |
527 | | - | |
528 | | - | |
529 | | - | |
530 | | - | |
531 | | - | |
532 | | - | |
533 | | - | |
534 | | - | |
535 | | - | |
536 | | - | |
537 | | - | |
538 | | - | |
539 | | - | |
540 | | - | |
541 | | - | |
542 | | - | |
543 | | - | |
544 | 507 | | |
545 | 508 | | |
546 | 509 | | |
| |||
0 commit comments