Skip to content

Remove ops that you can't test for correctness against #104

@PaliC

Description

@PaliC

While testing #92 I came across these aten ops which have errors. I will just stick the logs here. This is running opinfo against aten and torchbench --topn 3 against aten

Ops are:
empty_like.default -> we should actually remove testing for this one
new_empty.default -> we should actually remove testing for this one
bernoulli.default -> We either need a custom function or remove correctness here as well
new_empty_strided -> remove this one as well

opinfo against aten

[2025-08-21 11:45:56][WARNING][eval.py] 

Exception raised for empty_like.default:

    args: ((T([], bf16),), {})

    exc: Scalars are not close!



Expected 2.369355800876173e-38 but got -6.71875.

Absolute difference: 6.71875 (up to 0.01 allowed)

Relative difference: 2.8356863910078207e+38 (up to 0.01 allowed)



[2025-08-21 11:45:56][WARNING][eval.py] 

Exception raised for new_empty.default:

    args: ((T([5], bf16), (10,),), {'dtype': torch.float64})

    exc: Tensor-likes are not close!



Mismatched elements: 9 / 10 (90.0%)

Greatest absolute difference: 223118.2170910577 at index (3,) (up to 0.01 allowed)

Greatest relative difference: 59994.43869459766 at index (4,) (up to 0.01 allowed)



[2025-08-21 11:45:56][WARNING][eval.py] 

Exception raised for new_empty.default:

    args: ((T([5], bf16), (2, 2, 2),), {'dtype': torch.float64, 'device': 'cpu'})

    exc: Tensor-likes are not close!



Mismatched elements: 1 / 8 (12.5%)

Greatest absolute difference: nan at index (1, 1, 0) (up to 0.01 allowed)

Greatest relative difference: nan at index (1, 1, 0) (up to 0.01 allowed)



[2025-08-21 11:45:56][WARNING][eval.py] 

Exception raised for new_empty.default:

    args: ((T([5], bf16), (7, 2),), {'device': 'cuda'})

    exc: Tensor-likes are not close!



Mismatched elements: 13 / 14 (92.9%)

Greatest absolute difference: 1.05403626228257e+36 at index (6, 1) (up to 0.01 allowed)

Greatest relative difference: 2.03797651697492e+35 at index (6, 1) (up to 0.01 allowed)



[2025-08-21 11:45:56][WARNING][eval.py] 

Exception raised for new_empty_strided.default:

    args: ((T([5], bf16), (2, 2, 2), (20, 10, 5),), {'dtype': torch.float64, 'device': 'cpu'})

    exc: Tensor-likes are not close!



Mismatched elements: 1 / 8 (12.5%)

Greatest absolute difference: nan at index (0, 0, 0) (up to 0.01 allowed)

Greatest relative difference: nan at index (0, 0, 0) (up to 0.01 allowed)



[2025-08-21 11:45:56][WARNING][eval.py] 

Exception raised for bernoulli.default:

    args: ((T([], bf16),), {})

    exc: Scalars are not close!


Expected 1.0 but got 0.0.

Absolute difference: 1.0 (up to 0.01 allowed)

Relative difference: 1.0 (up to 0.01 allowed)

torchbench --topn 3 against aten

Exception raised for new_empty.default:
    args: ((T([64, 3136, 128], f16), [64, 1, 1],), {'dtype': torch.float16, 'layout': torch.strided, 'device': 'cuda', 'pin_memory': False})
    exc: Tensor-likes are not close!

Mismatched elements: 56 / 64 (87.5%)
Greatest absolute difference: 44224.0 at index (58, 0, 0) (up to 0.01 allowed)
Greatest relative difference: inf at index (1, 0, 0) (up to 0.01 allowed)

[2025-08-21 11:47:19][WARNING][eval.py] 
Exception raised for new_empty.default:
    args: ((T([64, 16, 196, 128], f16), [64, 1, 1, 1],), {'dtype': torch.float16, 'layout': torch.strided, 'device': 'cuda', 'pin_memory': False})
    exc: Tensor-likes are not close!

Mismatched elements: 16 / 64 (25.0%)
Greatest absolute difference: inf at index (58, 0, 0, 0) (up to 0.01 allowed)
Greatest relative difference: inf at index (58, 0, 0, 0) (up to 0.01 allowed)

[2025-08-21 11:47:19][WARNING][eval.py] 
Exception raised for new_empty_strided.default:
    args: ((T([128, 16, 112, 112], f16), [128, 16, 112, 112], [200704, 12544, 112, 1],), {})
    exc: Tensor-likes are not close!

Mismatched elements: 5140280 / 25690112 (20.0%)
Greatest absolute difference: 0.99951171875 at index (0, 0, 17, 3) (up to 0.01 allowed)
Greatest relative difference: inf at index (0, 0, 0, 0) (up to 0.01 allowed)

[2025-08-21 11:47:19][WARNING][eval.py] 
Exception raised for new_empty_strided.default:
    args: ((T([1, 1024, 128, 128], f16), [1, 1024, 128, 128], [16777216, 16384, 128, 1],), {})
    exc: Tensor-likes are not close!

Mismatched elements: 5140280 / 16777216 (30.6%)
Greatest absolute difference: 0.99951171875 at index (0, 544, 14, 115) (up to 0.01 allowed)
Greatest relative difference: inf at index (0, 544, 0, 0) (up to 0.01 allowed)

[2025-08-21 11:47:19][WARNING][eval.py] 
Exception raised for new_empty_strided.default:
    args: ((T([24, 4, 256, 513], f16), [24, 4, 256, 513], [525312, 131328, 513, 1],), {})
    exc: Tensor-likes are not close!

Mismatched elements: 5139725 / 12607488 (40.8%)
Greatest absolute difference: 0.99951171875 at index (0, 3, 161, 466) (up to 0.01 allowed)
Greatest relative difference: 1.0 at index (0, 3, 158, 98) (up to 0.01 allowed)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions