Count test totals correctly for dashboards (#280)

* Account test totals correctly for dashboards

* Add blurb to the dev guide on skipping tests

* Remove extra newline

* Default to 0 if "skipped" isn't found

Co-authored-by: Mathew Odden <[email protected]>

---------

Co-authored-by: Mathew Odden <[email protected]>

Author: charleshofer

.github/workflows/rocm-ci.yml

@@ -60,6 +60,7 @@ jobs:
           path: ./dist/*.whl
       - name: Run tests
         env:
+          ROCM_TEST_INCLUDE_SKIPS: "1"
           GPU_COUNT: "8"
           GFX: "gfx90a"
         run: |

rocm-downstream-dev-guide.md

@@ -30,7 +30,7 @@ This guide lays out how to do some dev operations, what branches live in this re
      If upstream reviewers request some changes to the new PR before merging, you can add
      or modify commits on the new `-upstream` feature branch.
   b. If this is an urgent change that we want in `rocm-main` right now but also want upstream,
-     add the `open-upstream` label, merge your PR, and then follow the link that 
+     add the `open-upstream` label, merge your PR, and then follow the link that
   c. If this is a change that we only want to keep in `rocm/jax` and not push into upstream,
      squash and merge your PR.
 
@@ -63,3 +63,7 @@ development tasks. These all live in `.github/workflows`.
 | ROCm Open Upstream PR      | `rocm-open-upstream-pr.yml`      | Add the `open-upstream` label to a PR                | Copies changes from a PR aimed at `rocm-main` into a new PR aimed at upstream's `main` |
 | ROCm Nightly Upstream Sync | `rocm-nightly-upstream-sync.yml` | Runs nightly, can be triggered manually via Actions  | Opens a PR that merges changes from upstream `main` into our `rocm-main` branch        |
 
+# Test Guidlines
+
+Use `pytest.mark.xfail` or `pytest.mark.xfail(run=False)` to skip failing tests that we will fix later. Use these for tests that we do eventually want to pass, but that we will skip for now to keep CI green. Continue to use `unittest.skip` or `self.skip` for tests that aren't applicable to AMD hardware.
+

tests/conftest.py

@@ -0,0 +1,14 @@
+import os
+import pytest
+
+
+INCLUDE_SKIPS = os.getenv("ROCM_TEST_INCLUDE_SKIPS", default=False)
+
+@pytest.hookimpl(optionalhook=True)
+def pytest_json_modifyreport(json_report):
+    """Get rid of skipped tests in reporting. We only care about xfails."""
+    if (not INCLUDE_SKIPS
+            and "summary" in json_report
+            and "total" in json_report["summary"]):
+        json_report["summary"]["unskipped_total"] = json_report["summary"]["total"] - json_report["summary"].get("skipped", 0)
+        del json_report["summary"]["total"]