[Bug] Shuffle read records inconsistent with the shuffle write records

[zuston]

Code of Conduct

• I agree to follow this project's Code of Conduct

Search before asking

• I have searched in the issues and found no similar issues.

Describe the bug

In our production spark job, we found a strange case that the shuffle read records number is inconsistent with the shuffle write records

Affects Version(s)

master

Uniffle Server Log Output

Uniffle Engine Log Output

Uniffle Server Configurations

Uniffle Engine Configurations

Additional context

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

[zuston]

cc @jerqi @yl09099

[yl09099]

Can this problem be stably reproduced?

[zuston]

Can this problem be stably reproduced?

No. that's hard to dig

[zuston]

apache/spark#50230

the above row based checksum looks a good design, I could follow this thought to implement in uniffle

[roryqi]

apache/spark#50230

the above row based checksum looks a good design, I could follow this thought to implement in uniffle

It's a good design. But seems that it can't solve the issue above.

[zuston]

apache/spark#50230
the above row based checksum looks a good design, I could follow this thought to implement in uniffle

It's a good design. But seems that it can't solve the issue above.

When initializing the shuffle reader, we could get the partition's checksum sum. And when reading, once iterator.next one row, we could calculate the checksum of this row and aggerate to compare with the initial checksum sum.

[roryqi]

apache/spark#50230
the above row based checksum looks a good design, I could follow this thought to implement in uniffle

It's a good design. But seems that it can't solve the issue above.

When initializing the shuffle reader, we could get the partition's checksum sum. And when reading, once iterator.next one row, we could calculate the checksum of this row and aggerate to compare with the initial checksum sum.

How to caculate the reduce partition checksum?