-
Notifications
You must be signed in to change notification settings - Fork 810
Open
Labels
Description
Some of the trig functions have tolerance issues for fp16. It sounds like we just need to adjust the tolerance for fp16 as it is currently defaulted to .0008, but that is intended for floats.
DxilConf_SM69_Vectorized::Asin_HLSLHalf_t
Error: Mismatch at Index: 5 Actual Value:-1.046875, Expected Value:-1.0458984375 diff = 0.0009765625
Error: Mismatch at Index: 8 Actual Value:-0.927734375, Expected Value:-0.9267578125 diff = 0.0009765625
DxilConf_SM69_Vectorized::Tan_HLSLHalf_t
Error: Mismatch at Index: 0 Actual Value:-1.5556640625, Expected Value:-1.5576171875 diff = 0.001953125
DxilConf_SM69_Vectorized::Acos_HLSLHalf_t
Error: Mismatch at Index: 9 Actual Value:2.703125, Expected Value:2.701171875 diff = 0.001953125
Maybe the tolerance for all of these should just be set to 0.003?
Metadata
Metadata
Assignees
Labels
Type
Projects
Status
No status