Detector datasets splits (#103)

* Sort categories dict in loaded dataset by key

* Add types to licenses in COCO schema

* Add image width and height as required in COCO file (for consistency with schema)

* Convert to torch datasets

* Add examples infrastructure and  proto example

* Add subsetsum notebook

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Dataset split with subset sum

* Add time estimate

* Draft detectors datasets module

* Split by video-pair

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update .gitignore for exampels

* Delete execution times file. Do not mock torch in docs build

* Add torch_dataset_to_annotations_dataset (WIP)

* Rename example

* Polish annotations as torch dataset example

* Select top 2 categories. Make sharp bits more explicit

* Simplify example. Add bbox conversion using torch utils

* Add note to example

* Use Counter rather than itertools (itertools counts consecutive instances). Use tolerance in number of samples. Use moren intuitive variable names. Sort by count if not shuffled

* Add dataset splits example (WIP)

* Rrename approximate_subset_sum to _approximate_subset_sum

* Define examples order in doc config

* Sort examples by filename

* Update example title

* Add loguru dependency and clean up Sphinx gallery configuration

* Improved approximate subset sum (WIP)

* Extend and filter list in one step

* Update example

* Refactor dataset splitting logic to handle empty subsets and maintain input order in return values.

* Updates to example

* Reduce example

* Move SubsetDict type definition

* Remove torch conversion bits. Rename examples and set order in docs config.

* Add some tests (WIP)

* More tests (WIP)

* Add docstring. Fix group ids as int

* Simplify casting group IDs as int

* Update docstring example

* Rename module. Update docstrings

* Add check and warning to random split. Return subsets in order of fractions. Rename input parameter

* Rename ds input parameter in saving function

* Expand tests

* Add test for warning

* Add pytest-loguru to development dependencies

* Review and small edits to coco_bbox_ethology_and_movement.py

* Edit example intro

* Add sklearn groupkfold version and preliminary tests

* Small edits to COCO example

* Refactor dataset splitting functions and update tests to use new names.

The functions `split_dataset_group_by_sklearn` and `split_dataset_group_by` have been renamed to `_split_dataset_group_by_kfold` and `_split_dataset_group_by_apss`, respectively. Corresponding test functions have also been updated to reflect these changes, and new tests for k-fold splitting with seed functionality have been added.

* Add docstring with examples

* Add wrapper function to split group by

* Fix tests

* Add test to check method dispatch

* Add test to check auto delegates correctly

* Add test for unknown method

* Fix docstring example

* Small edits to API

* Add logger info for method and test

* Fix movement link in docs

* Review example

* Fix movement link

* Fix movement link

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

Author: sfmig

MANIFEST.in

@@ -9,7 +9,7 @@ recursive-exclude * *.py[co]
 recursive-exclude docs *
 recursive-exclude tests *
 recursive-exclude examples *
-recursive-include docs *.md *.rst *.py
+recursive-include docs *
 
 # Include json schemas
 recursive-include ethology/io/annotations/json_schemas/schemas *.json

docs/source/conf.py

@@ -1,13 +1,14 @@
-# Configuration file for the Sphinx documentation builder.
 """Sphinx configuration for ethology documentation."""
 
 import os
 import sys
 from importlib.metadata import version as get_version
 
+from sphinx_gallery import sorting
+
 # Used when building API docs, put the dependencies
 # of any class you are documenting here
-autodoc_mock_imports: list[str] = ["cv2", "torch"]
+autodoc_mock_imports: list[str] = ["cv2"]
 
 # Add the module path to sys.path here.
 # If the directory is relative to the documentation root,
@@ -179,7 +180,8 @@
     "matplotlib": ("https://matplotlib.org/stable/", None),
     "numpy": ("https://numpy.org/doc/stable/", None),
     "pandera": ("https://pandera.readthedocs.io/en/stable/", None),
-    "movement": ("https://movement.neuroinformatics.dev/", None),
+    "movement": ("https://movement.neuroinformatics.dev/latest/", None),
+    "sklearn": ("https://scikit-learn.org/stable/", None),
 }
 
 
@@ -207,6 +209,13 @@
 
 sphinx_gallery_conf = {
     "examples_dirs": ["../../examples"],
+    "within_subsection_order": sorting.ExplicitOrder(
+        [
+            "coco_bbox_ethology_and_movement.py",
+            "approximate_subset_sum_split.py",
+            "*",
+        ]
+    ),
     "filename_pattern": "/*.py",  # which files to execute before inclusion
     "gallery_dirs": ["examples"],  # output directory
     "run_stale_examples": True,  # re-run examples on each build

ethology/__init__.py

@@ -1,10 +1,14 @@
 from importlib.metadata import PackageNotFoundError, version
-
 import xarray as xr
+from pathlib import Path
 
 # Set xarray attributes collapsed by default
 xr.set_options(display_expand_attrs=False)
 
+# Set cache directory for ethology package
+ETHOLOGY_CACHE_DIR = Path.home() / ".ethology"
+ETHOLOGY_CACHE_DIR.mkdir(parents=True, exist_ok=True)
+
 try:
     __version__ = version("ethology")
 except PackageNotFoundError: