Add unified Hierarchical MetricCollection (torchmetrics) + unit tests

[Jyc323]

Summary

This PR introduces a hierarchical metrics module and a consistent entrypoint for training/eval. It includes leaf accuracy, full-path accuracy, prediction path-consistency ratios, and label-count–weighted precision into a single MetricCollection that plugs into existing OTX/torchmetrics flows.

• LeafAccuracy
  Macro-averaged correctness at the leaf level (final decision); mitigates class-imbalance bias.

• FullPathAccuracy
  Strict correctness across all levels; prevents inflated scores from partial matches.

• InconsistentPathRatio (predictions)
  Detects taxonomy-violating predictions (child not under parent); highlights model–hierarchy mismatches.

• WeightedHierarchicalPrecision
  Per-level macro precision aggregated with label-count weights; balances coarse vs. fine levels and stays robust to imbalance.

How to test

pytest -q lib/tests/unit/metrics/test_hier_metric_collection.py

Checklist

• I have added unit tests to cover my changes.​
• I have added integration tests to cover my changes.​
• I have ran e2e tests and there is no issues.
• I have added the description of my changes into CHANGELOG in my target branch (e.g., CHANGELOG in develop).​
• I have updated the documentation in my target branch accordingly (e.g., documentation in develop).
• I have linked related issues.

License

• I submit my code changes under the same Apache License that covers the project.
  Feel free to contact the maintainers if that's a concern.
• I have updated the license header for each file (see an example below).

# Copyright (C) 2025 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

[sovrasov]

Thanks @Jyc323 for extending OTX h-cls metrics collection!
Considering #4706, it makes sense to create a chapter for h-cls in https://open-edge-platform.github.io/training_extensions/latest/guide/tutorials/advanced/index.html
As in the loss PR, there should be a way how OTX users can quickly utilize new metrics by modifying model recipe. Also, it might be an idea to extend OTX default h-cls metrics collection to make the default metrics more expressive right out of the box

[Jyc323]

Hi @sovrasov, thanks for the review and comments. Per your feedback, I removed the typing imports and now use the built-in generics (e.g., list, dict) in this push—please take another look. I also fixed all issues reported by tox -vv -e pre-commit.

As an extension, I integrated the current hierarchical accuracy calculation into hier_metric_collection_callable. When using otx.engine for training or testing, users can pass the metric parameter to enable it. I added a corresponding unit test at tests/unit/metrics/test_hier_metric_collection_from_engine.py. I’d appreciate your thoughts and any suggestions.

[sovrasov]

Hi @sovrasov, thanks for the review and comments. Per your feedback, I removed the typing imports and now use the built-in generics (e.g., list, dict) in this push—please take another look. I also fixed all issues reported by tox -vv -e pre-commit.

As an extension, I integrated the current hierarchical accuracy calculation into hier_metric_collection_callable. When using otx.engine for training or testing, users can pass the metric parameter to enable it. I added a corresponding unit test at tests/unit/metrics/test_hier_metric_collection_from_engine.py. I’d appreciate your thoughts and any suggestions.

Thanks for the update! I see there is a way how users can utilize the new metrics, but it's hard hard to find it without a careful review of OTX source code. That I'm suggesting here is to add a tutorial on how to use the new h-cls features via both OTX recipe and API. You can check other tutorials for instance, docs/source/guide/tutorials/advanced/peft.rst to have an idea how to write one. The docs can be built by running cd lib && tox -e build-doc

[sovrasov]

@kprokofi will temporally cover me, as I'm off next week

[Jyc323]

@kprokofi I added the documentation under docs/source/guide/tutorials/advanced as suggestion. Please have a look and give me any feedback, since I'm new to the documentation tool.

[Jyc323]

Hi @sovrasov , thanks for the reviewing. I've reviewed the errors from the Unit-Test-with-Python3.12 run, but they don't appear to be caused by my changes. Could you please advise on what I should do?

=========================== short test summary info ============================
ERROR tests/unit/backend/native/models/detection/test_yolox.py - OSError: [Er...
ERROR tests/unit/backend/native/models/instance_segmentation/test_maskrcnn.py
ERROR tests/unit/backend/native/models/segmentation/test_dino_v2_seg.py - OSE...
ERROR tests/unit/backend/native/models/segmentation/test_segnext.py - OSError...
!!!!!!!!!!!!!!!!!!! Interrupted: 4 errors during collection !!!!!!!!!!!!!!!!!!!!
================== 22 warnings, 4 errors in 85.47s (0:01:25) ===================

As for 'Required Check lib-lint-and-test', I don't think the log provide enough information, could you please tell me where I can find detailed information?

Error: Required status checks failed. They must succeed before this pull request can be merged.
Error: Process completed with exit code 1.

Thanks for your help

[sovrasov]

Hi @sovrasov , thanks for the reviewing. I've reviewed the errors from the Unit-Test-with-Python3.12 run, but they don't appear to be caused by my changes. Could you please advise on what I should do?

=========================== short test summary info ============================
ERROR tests/unit/backend/native/models/detection/test_yolox.py - OSError: [Er...
ERROR tests/unit/backend/native/models/instance_segmentation/test_maskrcnn.py
ERROR tests/unit/backend/native/models/segmentation/test_dino_v2_seg.py - OSE...
ERROR tests/unit/backend/native/models/segmentation/test_segnext.py - OSError...
!!!!!!!!!!!!!!!!!!! Interrupted: 4 errors during collection !!!!!!!!!!!!!!!!!!!!
================== 22 warnings, 4 errors in 85.47s (0:01:25) ===================

As for 'Required Check lib-lint-and-test', I don't think the log provide enough information, could you please tell me where I can find detailed information?

Error: Required status checks failed. They must succeed before this pull request can be merged.
Error: Process completed with exit code 1.

Thanks for your help

Indeed, that can be unrelated CI errors already fixed in develop. Could you merge changes from develop and resolve conflicts first? Then, we can run the CI one more time

[Jyc323]

Hi @sovrasov, I already merged and resolved the conflicting, please let me know if there are any issues. Thanks a lot

[sovrasov]

Unfortunately, CI is still not in the best shape, ETA for a fix is next week
BTW did you double-check if my latest review comment in this PR makes sense?

[Jyc323]

Hi @sovrasov, I think your latest review comment makes sense. I already fixed it and pushed the modification. Please let me know anything else I need to do from my side. Have a great day!