-
Notifications
You must be signed in to change notification settings - Fork 2.7k
[CPU] fix wrong result in ScatterNDUpadte when there are conflict indices #31650
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[CPU] fix wrong result in ScatterNDUpadte when there are conflict indices #31650
Conversation
Signed-off-by: HU Yuan2 <[email protected]>
Signed-off-by: HU Yuan2 <[email protected]>
Signed-off-by: HU Yuan2 <[email protected]>
582bdec
to
e6e9fb9
Compare
Signed-off-by: HU Yuan2 <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR fixes a thread safety issue in the ScatterNDUpdate operation in the Intel CPU plugin where conflicting indices could produce incorrect results due to multi-threaded execution. The fix ensures that when multiple indices point to the same location, only the last update operation is applied, maintaining the expected behavior.
- Introduces mutex-based synchronization to handle conflicting indices in multi-threaded execution
- Adds tracking of the last valid update index to ensure correct ordering
- Includes a new test case with conflicting indices to verify the fix
Reviewed Changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
File | Description |
---|---|
scatter_update.cpp | Implements mutex-based synchronization and index tracking to handle conflicting indices correctly |
scatter_ND_update.cpp | Adds test case with conflicting indices to validate the fix |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
You can also share your feedback on Copilot code review for a chance to win a $100 gift card. Take the survey.
@@ -1049,6 +1050,10 @@ void ScatterUpdate::scatterNDUpdate(const MemoryPtr& mem_data, | |||
std::vector<size_t> srcBlockND = getBlockND(srcDataDim); | |||
|
|||
size_t k = indicesDim[indicesRank - 1]; | |||
// mutexVec and updatedIdxVec are used for idxValue conflicts, | |||
// only the last idxValues of update vec is valid when there are idxValue conflicts, | |||
std::vector<std::mutex> mutexVec(srcBlockND[0] / srcBlockND[k]); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Creating a vector of mutexes equal to the number of slices may cause performance degradation and excessive memory usage for large tensors. Consider using a smaller pool of mutexes with a hash-based mapping to reduce overhead.
std::vector<std::mutex> mutexVec(srcBlockND[0] / srcBlockND[k]); | |
constexpr size_t NUM_MUTEXES = 1024; // or 256, tune as needed | |
std::vector<std::mutex> mutexVec(NUM_MUTEXES); |
Copilot uses AI. Check for mistakes.
cpu_memcpy(dstData + dstOffset, update + updateOffset, sizeToUpdate); | ||
updatedIdxVec[sliceIdx] = tupleIdx; | ||
} | ||
mutexVec[sliceIdx].unlock(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Using manual mutex lock/unlock is error-prone and can lead to deadlocks if an exception occurs between lock and unlock. Use std::lock_guard or std::unique_lock for RAII-based exception-safe locking.
mutexVec[sliceIdx].unlock(); | |
std::lock_guard<std::mutex> lock(mutexVec[sliceIdx]); | |
if (static_cast<int64_t>(tupleIdx) > updatedIdxVec[sliceIdx]) { | |
cpu_memcpy(dstData + dstOffset, update + updateOffset, sizeToUpdate); | |
updatedIdxVec[sliceIdx] = tupleIdx; | |
} |
Copilot uses AI. Check for mistakes.
cpu_memcpy(dstData + dstOffset, update + updateOffset, sizeToUpdate); | ||
updatedIdxVec[sliceIdx] = tupleIdx; | ||
} | ||
mutexVec[sliceIdx].unlock(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Manual unlock should be replaced with RAII-based locking (std::lock_guard) to ensure the mutex is properly unlocked even if an exception occurs.
mutexVec[sliceIdx].unlock(); | |
{ | |
std::lock_guard<std::mutex> lock(mutexVec[sliceIdx]); | |
if (static_cast<int64_t>(tupleIdx) > updatedIdxVec[sliceIdx]) { | |
cpu_memcpy(dstData + dstOffset, update + updateOffset, sizeToUpdate); | |
updatedIdxVec[sliceIdx] = tupleIdx; | |
} | |
} |
Copilot uses AI. Check for mistakes.
Hi, @tiger100256-hu : thanks for root cause this puzzling CI testcase issue. and the method implemented in this pr looks like could solve this kind of CI failure phenomenon. |
@liubo-intel thanks very much, I forget to see this spec. from the spec, openvino ScatterNDUpdate don't support conflict indices so, the convert maybe is not right here
|
close this pr, it 's not a correct way to fix this issue. |
Details:
Tickets: