Experimental datumaro implementation #4920

AlbertvanHouten · 2025-10-20T08:10:13Z

This pull request introduces the new experimental dataset into the dataset classes of OTX. A lot of logic has been migrated to datumaro such as management of image color channels and the tiling implementation.

Summary

How to test

Checklist

The PR title and description are clear and descriptive
I have manually tested the changes
All changes are covered by automated tests
All related issues are linked to this PR (if applicable)
Documentation has been updated (if applicable)

Signed-off-by: Albert van Houten <[email protected]> Co-authored-by: Leonardo Lai <[email protected]>

…ing_extensions into feature/datumaro

Signed-off-by: Albert van Houten <[email protected]>

…4751) Signed-off-by: Albert van Houten <[email protected]> Co-authored-by: Grégoire Payen de La Garanderie <[email protected]>

…ing_extensions into feature/datumaro # Conflicts: # library/src/otx/data/dataset/base_new.py # library/src/otx/data/dataset/classification_new.py # library/src/otx/data/dataset/detection_new.py # library/src/otx/data/dataset/instance_segmentation_new.py # library/src/otx/data/dataset/keypoint_detection_new.py # library/src/otx/data/dataset/segmentation_new.py # library/src/otx/data/entity/sample.py

Signed-off-by: Albert van Houten <[email protected]>

…ing_extensions into feature/datumaro

Signed-off-by: Albert van Houten <[email protected]>

Signed-off-by: Albert van Houten <[email protected]> Co-authored-by: Albert van Houten <[email protected]>

Signed-off-by: Albert van Houten <[email protected]>

…orm/training_extensions into feature/datumaro

Signed-off-by: Albert van Houten <[email protected]>

…ing_extensions into feature/datumaro Signed-off-by: Albert van Houten <[email protected]> # Conflicts: # library/pyproject.toml # library/src/otx/data/dataset/anomaly.py # library/src/otx/data/dataset/base.py # library/src/otx/data/dataset/classification.py # library/src/otx/data/dataset/detection.py # library/src/otx/data/dataset/instance_segmentation.py # library/src/otx/data/dataset/keypoint_detection.py # library/src/otx/data/dataset/segmentation.py # library/src/otx/data/dataset/tile.py # library/src/otx/data/factory.py # library/src/otx/data/module.py # library/src/otx/data/transform_libs/torchvision.py # library/tests/unit/data/samplers/test_class_incremental_sampler.py # library/tests/unit/data/utils/test_utils.py

Copilot

Pull Request Overview

This PR implements an experimental Datumaro dataset integration for OTX, transitioning from legacy Datumaro components to the new experimental Dataset API. The changes introduce a new sample-based architecture while maintaining compatibility with existing OTX functionality.

Key changes:

Migration from legacy Datumaro components to experimental Dataset API with schema-based conversion
Introduction of new OTXSample-based data entities with PyTree registration for TorchVision compatibility
Replacement of legacy polygon handling with numpy ragged arrays for better performance
Comprehensive test updates and new test implementations for the updated dataset architecture

Reviewed Changes

Copilot reviewed 59 out of 59 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
pyproject.toml	Updates Datumaro dependency to experimental branch version
library/src/otx/data/dataset/*.py	Implements new dataset classes using experimental Datumaro with sample-based architecture
library/src/otx/data/entity/sample.py	Adds new OTXSample classes with PyTree registration for transform compatibility
library/tests/unit/types/test_label.py	Updates label tests to use new hierarchical label categories
library/tests/unit/data/transform_libs/test_torchvision.py	Converts polygon handling from Datumaro objects to numpy arrays

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-10-29T09:25:48Z

library/tests/unit/data/transform_libs/test_torchvision.py

+    fake_polygons = np.array(
+        [
+            np.array([[10, 10], [50, 10], [50, 50], [10, 50]]),  # Rectangle polygon for first object
+            np.array([[60, 60], [100, 60], [100, 100], [60, 100]]),  # Rectangle polygon for second object
+        ]
+    )


The polygon data structure has changed from Datumaro Polygon objects to numpy ragged arrays, but the test doesn't validate that the new format maintains the same geometric properties. Consider adding assertions to verify that the polygon areas and vertex coordinates are equivalent between the old and new formats.

Copilot · 2025-10-29T09:25:48Z

library/tests/unit/data/transform_libs/test_torchvision.py

-            p = np.asarray(polygon.points)
-            p[0::2] = width - p[0::2]
-            return p.tolist()
+        def revert_hflip(polygon: np.ndarray, width: int) -> np.ndarray:


The function modifies the input polygon array in-place, which could cause issues if the original array is used elsewhere. The function should create a copy before modification to avoid unintended side effects.

Suggested change

def revert_hflip(polygon: np.ndarray, width: int) -> np.ndarray:

def revert_hflip(polygon: np.ndarray, width: int) -> np.ndarray:

polygon = polygon.copy()

library/src/otx/data/transform_libs/utils.py

# Conflicts: # library/tests/unit/data/dataset/test_base.py # library/tests/unit/data/test_tiling.py

Signed-off-by: Albert van Houten <[email protected]>

…atumaro

Signed-off-by: Albert van Houten <[email protected]>

…ing_extensions into feature/datumaro

Signed-off-by: Albert van Houten <[email protected]>

Copilot

Pull request overview

Copilot reviewed 81 out of 82 changed files in this pull request and generated 4 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

library/src/otx/types/label.py

library/src/otx/backend/native/models/__init__.py

library/src/otx/data/dataset/base.py

Copilot · 2025-12-04T08:53:28Z

library/src/otx/backend/native/callbacks/gpu_mem_monitor.py

            pl_module (LightningModule): pl module.
            batch_size (int): batch size.
        """
        device = trainer.strategy.root_device


The addition of 'mps' device type is good, but consider documenting why MPS (Apple Metal Performance Shaders) devices should skip GPU memory monitoring, similar to CPU and XPU.

Suggested change

device = trainer.strategy.root_device

device = trainer.strategy.root_device

# Skip GPU memory monitoring for CPU, XPU, and MPS (Apple Metal Performance Shaders) devices.

# MPS devices do not support the same GPU memory monitoring APIs as CUDA devices,

# so memory stats are unavailable or unreliable, similar to CPU and XPU.

Signed-off-by: Albert van Houten <[email protected]>

mramotowski · 2025-12-04T09:22:43Z

.github/workflows/lib-lint-and-test.yaml

-        uses: dtolnay/rust-toolchain@b9ed5a8fb8afb645c9cfaffecc60fc946d28f857 # stable
+      # TODO remove after switching to an official Datumaro release with pre-built wheels
+      - name: Installing Rust toolchain
+        uses: dtolnay/rust-toolchain@6d653acede28d24f02e3cd41383119e8b1b35921


Why this sha? What version is it? Pls add info in comment. I checked latest versions and can't see this commit:
commit 0d231036f5e097127fd969f80fb8e44d7394f7f9 (HEAD -> 1.91.1, origin/1.91.1, origin/1.91)
commit a4a18948872904a1ea9bf180c73ce0a3ce1ceaab (HEAD -> 1.91.0, origin/1.91.0)
The commit you provide looks like commit from master branch:
commit 6d653acede28d24f02e3cd41383119e8b1b35921
Shouldn't we use some of release versions?

It's probably taken from #5054, which I copied from the Datumaro repo [ref].

The maintainer of this action dtolnay/rust-toolchain keeps updating his project but doesn't make official GH releases (actually there's one release - v1 - but it's from 2022).

Co-authored-by: Copilot <[email protected]>

Signed-off-by: Albert van Houten <[email protected]>

…gic. Signed-off-by: Albert van Houten <[email protected]>

- Remove data module/loader dependency from simple model tests. Added a fixture to generate a sample batch - Update rtdetr model to stack OTXDataBatch images if they are provided as a list. Signed-off-by: Albert van Houten <[email protected]>

Signed-off-by: Albert van Houten <[email protected]>

codecov-commenter · 2025-12-05T13:13:32Z

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 74.75410% with 154 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
library/src/otx/data/dataset/classification.py	66.12%	21 Missing ⚠️
...ary/src/otx/backend/native/models/detection/ssd.py	18.18%	18 Missing ⚠️
library/src/otx/backend/native/tools/tile_merge.py	11.76%	15 Missing ⚠️
library/src/otx/data/transform_libs/torchvision.py	67.39%	15 Missing ⚠️
library/src/otx/data/transform_libs/utils.py	76.92%	15 Missing ⚠️
...ry/src/otx/data/utils/structures/mask/mask_util.py	0.00%	13 Missing ⚠️
library/src/otx/data/entity/tile.py	42.85%	12 Missing ⚠️
library/src/otx/data/entity/sample.py	93.93%	8 Missing ⚠️
...native/models/instance_segmentation/rotated_det.py	25.00%	6 Missing ⚠️
library/src/otx/data/factory.py	78.57%	6 Missing ⚠️
... and 11 more

📢 Thoughts on this report? Let us know!

Move memory cleanup to model tests only Signed-off-by: Albert van Houten <[email protected]>

Signed-off-by: Albert van Houten <[email protected]>

… flag Signed-off-by: Albert van Houten <[email protected]>

Signed-off-by: Albert van Houten <[email protected]>

AlbertvanHouten and others added 16 commits September 4, 2025 10:10

Classification Dataset refactor (#4606)

9bd3fa7

Signed-off-by: Albert van Houten <[email protected]> Co-authored-by: Leonardo Lai <[email protected]>

Merge branch 'develop' of https://github.com/open-edge-platform/train…

373a046

…ing_extensions into feature/datumaro

Detection dataset updated to use new datumaro version (#4675)

9f26bb8

Signed-off-by: Albert van Houten <[email protected]>

Update the segmentation dataset to use new Datumaro dataset (#4676)

8b2fe8b

Instance segmentation support with the new Datumaro dataset (#4686)

3d84b50

Migrate OTX Keypoint detection to use the new experimental dataset (#…

d1ff0dc

…4751) Signed-off-by: Albert van Houten <[email protected]> Co-authored-by: Grégoire Payen de La Garanderie <[email protected]>

Move new dataset files to library folder

a3d82dd

Signed-off-by: Albert van Houten <[email protected]>

Update OTX classification task to use legacy converter (#4769)

c1b8084

Signed-off-by: Albert van Houten <[email protected]>

Merge branch 'develop' of https://github.com/open-edge-platform/train…

9fd7abc

…ing_extensions into feature/datumaro

Migrate anomaly task to use the new Datumaro dataset (#4770)

4751cb2

Merge branch 'develop' of https://github.com/open-edge-platform/train…

5ff3b59

…ing_extensions into feature/datumaro

Point datumaro deps to the dev branch

6088088

Signed-off-by: Albert van Houten <[email protected]>

Tilling support with the experimental Datumaro dataset (#4858)

7827ca5

Signed-off-by: Albert van Houten <[email protected]> Co-authored-by: Albert van Houten <[email protected]>

Use new datumaro for multi label and hierarchical classification (#4815)

64e58ca

Signed-off-by: Albert van Houten <[email protected]>

Merge branch 'feature/datumaro' of https://github.com/open-edge-platf…

fdb6afe

…orm/training_extensions into feature/datumaro

github-actions bot added DEPENDENCY Any changes in any dependencies (new dep or its version) should be produced via Change Request on PM TEST Any changes in tests BUILD labels Oct 20, 2025

AlbertvanHouten added 4 commits October 23, 2025 11:55

Replace old datasets with new one

bdaa084

Signed-off-by: Albert van Houten <[email protected]>

Update hierarchical label tests to use new datumaro categories

eb6fbda

Signed-off-by: Albert van Houten <[email protected]>

Cleanup remaining tests

a7d0c51

Signed-off-by: Albert van Houten <[email protected]>

AlbertvanHouten changed the title ~~Feature/datumaro~~ Experimental datumaro implementation Oct 29, 2025

AlbertvanHouten marked this pull request as ready for review October 29, 2025 09:24

AlbertvanHouten requested a review from a team as a code owner October 29, 2025 09:24

Copilot AI review requested due to automatic review settings October 29, 2025 09:24

Copilot AI reviewed Oct 29, 2025

View reviewed changes

leoll2 added 2 commits November 1, 2025 10:13

Merge branch 'develop' into feature/datumaro

3efe5aa

# Conflicts: # library/tests/unit/data/dataset/test_base.py # library/tests/unit/data/test_tiling.py

fix random generator

715bbb5

leoll2 and others added 8 commits December 1, 2025 11:37

Merge branch 'develop' into feature/datumaro

0a1d8c4

Fix Detection sample and pass arguments for legacy dataset conversion

1ab6125

Signed-off-by: Albert van Houten <[email protected]>

Merge remote-tracking branch 'origin/feature/datumaro' into feature/d…

b5b68ed

…atumaro

Fix transforms that were wrongly being applied to keypoints/labels.

5084fdb

Signed-off-by: Albert van Houten <[email protected]>

Add additional test cases

72829cd

Signed-off-by: Albert van Houten <[email protected]>

Revert lock file changes

069919d

Signed-off-by: Albert van Houten <[email protected]>

Merge branch 'develop' of https://github.com/open-edge-platform/train…

f881514

…ing_extensions into feature/datumaro

update lock

7d5f5a9

Signed-off-by: Albert van Houten <[email protected]>

AlbertvanHouten requested a review from Copilot December 4, 2025 08:51

Copilot AI reviewed Dec 4, 2025

View reviewed changes

Use uv for python version management

b407eb1

Signed-off-by: Albert van Houten <[email protected]>

AlbertvanHouten requested a review from a team as a code owner December 4, 2025 09:02

AlbertvanHouten added 2 commits December 4, 2025 10:08

Add rust compiler

75e8d42

Signed-off-by: Albert van Houten <[email protected]>

Remove forkserver logic

72a904a

Signed-off-by: Albert van Houten <[email protected]>

mramotowski reviewed Dec 4, 2025

View reviewed changes

AlbertvanHouten and others added 2 commits December 4, 2025 10:28

Update library/src/otx/data/dataset/base.py

96be71e

Co-authored-by: Copilot <[email protected]>

Fix formatting and fix a bug in label mapping

1cdeccd

Signed-off-by: Albert van Houten <[email protected]>

AlbertvanHouten force-pushed the feature/datumaro branch from 84157a6 to 1cdeccd Compare December 4, 2025 14:06

Test a fix for hanging of tests in CI

d32d5c0

Signed-off-by: Albert van Houten <[email protected]>

AlbertvanHouten force-pushed the feature/datumaro branch from e2687e7 to d32d5c0 Compare December 5, 2025 07:50

AlbertvanHouten added 4 commits December 5, 2025 10:08

Allow tests to be run from any directory by improving path finding lo…

6932802

…gic. Signed-off-by: Albert van Houten <[email protected]>

Fix failing detection tests:

4871fe4

- Remove data module/loader dependency from simple model tests. Added a fixture to generate a sample batch - Update rtdetr model to stack OTXDataBatch images if they are provided as a list. Signed-off-by: Albert van Houten <[email protected]>

Apply same fixes for datamodule to segmentation tests

b9e750f

Signed-off-by: Albert van Houten <[email protected]>

Reduce memory usage for detection models

a2631a5

Signed-off-by: Albert van Houten <[email protected]>

AlbertvanHouten added 5 commits December 5, 2025 14:15

Move multiprocessing start logic to conftest

b48a638

Move memory cleanup to model tests only Signed-off-by: Albert van Houten <[email protected]>

Attempt to fixt exit code 139

811b08f

Signed-off-by: Albert van Houten <[email protected]>

Separate model tests from other unit tests and run them with --forked…

977c14f

… flag Signed-off-by: Albert van Houten <[email protected]>

Revert timeout changes

e242492

Signed-off-by: Albert van Houten <[email protected]>

Swap order of GHA steps and re-enable cuda tests

d5819a3

Signed-off-by: Albert van Houten <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Experimental datumaro implementation #4920

Experimental datumaro implementation #4920

Uh oh!

AlbertvanHouten commented Oct 20, 2025 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Oct 29, 2025

Uh oh!

Copilot AI Oct 29, 2025

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Dec 4, 2025

Uh oh!

mramotowski Dec 4, 2025

Uh oh!

leoll2 Dec 4, 2025

Uh oh!

codecov-commenter commented Dec 5, 2025 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

	def revert_hflip(polygon: np.ndarray, width: int) -> np.ndarray:
	def revert_hflip(polygon: np.ndarray, width: int) -> np.ndarray:
	polygon = polygon.copy()

-        device = trainer.strategy.root_device
+        device = trainer.strategy.root_device
+        # Skip GPU memory monitoring for CPU, XPU, and MPS (Apple Metal Performance Shaders) devices.
+        # MPS devices do not support the same GPU memory monitoring APIs as CUDA devices,
+        # so memory stats are unavailable or unreliable, similar to CPU and XPU.

Experimental datumaro implementation #4920

Are you sure you want to change the base?

Experimental datumaro implementation #4920

Uh oh!

Conversation

AlbertvanHouten commented Oct 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

How to test

Checklist

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Oct 29, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 29, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Dec 4, 2025

Choose a reason for hiding this comment

Uh oh!

mramotowski Dec 4, 2025

Choose a reason for hiding this comment

Uh oh!

leoll2 Dec 4, 2025

Choose a reason for hiding this comment

Uh oh!

codecov-commenter commented Dec 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

AlbertvanHouten commented Oct 20, 2025 •

edited

Loading

codecov-commenter commented Dec 5, 2025 •

edited

Loading