Skip to content

Long Vector 16-bit trig function tolerances #7936

@alsepkow

Description

@alsepkow

Some of the trig functions have tolerance issues for fp16. It sounds like we just need to adjust the tolerance for fp16 as it is currently defaulted to .0008, but that is intended for floats.

DxilConf_SM69_Vectorized::Asin_HLSLHalf_t

    Error: Mismatch at Index: 5 Actual Value:-1.046875, Expected Value:-1.0458984375           diff = 0.0009765625

    Error: Mismatch at Index: 8 Actual Value:-0.927734375, Expected Value:-0.9267578125   diff = 0.0009765625

DxilConf_SM69_Vectorized::Tan_HLSLHalf_t

    Error: Mismatch at Index: 0 Actual Value:-1.5556640625, Expected Value:-1.5576171875   diff = 0.001953125

DxilConf_SM69_Vectorized::Acos_HLSLHalf_t

    Error: Mismatch at Index: 9 Actual Value:2.703125, Expected Value:2.701171875                   diff = 0.001953125

Maybe the tolerance for all of these should just be set to 0.003?

Metadata

Metadata

Assignees

Labels

bugBug, regression, crashneeds-triageAwaiting triage

Type

No type

Projects

Status

No status

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions