-
-
Notifications
You must be signed in to change notification settings - Fork 13
Description
We should add a "known issue" result type. It would correspond to when a test input for a platform/library matches criteria for an existing bug filed in the upstream project of the platform.
The benefit of having the result type is to better categorize / triage the results of our executor implementations, as we continue to update and improve them. Currently, we have test failures, and those failures could be the result of: the library (unknown bug), the library (known bug), or a deficiency in our executor. Sometimes, a known issue causes a systematic set of many test failures.
For the purposes of our project, it would help instill confidence in our self-assessment of correct executor implementation work to not include known bugs in the set of test failures.