Add exception handling for numerical errors (#13026)

Summary:

As titled. Instead of skipping, those tests will compare the errors to the existing values.
Remove all updated tolerances from the runtime update side and make it an expected numerical accuracy. We should open tasks to trace the (large) errors, will do that as a follow up.

On RT600, WW stage2, rnnt predictor and asr joiner are all showing flakiness. Opened T232829844 to track.

Timeouts are still skipped for now.

Reviewed By: abeakkas, zonglinpeng

Differential Revision: D79065198

Author: mcremon-meta

backends/cadence/aot/utils.py

@@ -29,6 +29,22 @@ class MemoryPlanningAlgoFailure(Exception):
     pass
 
 
+class TypeMismatchError(Exception):
+    pass
+
+
+class NumericalMismatchError(Exception):
+    def __init__(self, msg: str, rms_value: Optional[float] = None) -> None:
+        self.rms_value = rms_value
+        super().__init__(msg)
+
+
+class NumericalMismatchExpectedError(Exception):
+    def __init__(self, rms_expected_value: float) -> None:
+        self.rms_expected_value = rms_expected_value
+        super().__init__()
+
+
 # Get the output size of a 1D convolution given the input size and parameters
 def get_conv1d_output_size(
     in_size: torch.Size,