Merge pull request #120 from FocoosAI/feat/implement-rtmo

Key Changes
✨ Introduce keypoints models
add RTMO-S/M/L-COCO keypoint pretrained model
example:
from focoos import ModelManager
from PIL import Image

im = "https://public.focoos.ai/samples/federer.jpg"
model = ModelManager.get("rtmo-s-coco")
detections = model.infer(im,annotate=True, threshold=0.5)
Image.fromarray(detections.image) # visualise or save annotated image
📷 Unified Inference API
Standardize infer Method Signatures
consistent infer() method across FocoosModel, InferModel, and RemoteModel with unified parameters: infer(image, threshold=0.5, annotate=False) and use unified image loader for infer methods (with also remote image support)
add default threshold to 0.5
Remove dependency on external annotate_image() function calls
Streamlined workflow: get detections and visual annotations in a single call
example torch and exported model:

from focoos import ModelManager, RuntimeType
from PIL import Image


im = "https://public.focoos.ai/samples/motogp.jpg" # remote image, can also be local path, numpy array, or PIL image
model = ModelManager.get("fai-detr-l-obj365")
detections = model.infer(im,annotate=True, threshold=0.5) # annotatate param 
# Image.fromarray(detections.image) # visualise or save annotated image

# export model
model = model.export(RuntimeType.ONNX_CUDA32)
res = model.infer(im, annotate=True, threshold=0.5)

Image.fromarray(detections.image) # visualise or save annotated image
example with remote inference:

from focoos import FocoosHUB
from PIL import Image

hub = FocoosHUB()
model_ref = "fai-detr-l-obj365" # use any of pretrained model on app.focoos.ai or your own model reference
remote_model = hub.get_remote_model(model_ref)

im = "https://public.focoos.ai/samples/federer.jpg"

detections = remote_model.infer(im,annotate=True, threshold=0.5)

Image.fromarray(detections.image) # visualise or save annotated image
Enhanced FocoosDetections Structure
add new image field: stores annotated results as base64 string or numpy array
migrated from Pydantic to pure Python dataclasses for better performance,
Improved serialization and memory usage
add new keypoints field
add pprint and print_infer methods to unify detections prints.
⌨️ CLI
add new CLI command: focoos gradio to launch a Gradio interface for image and video inference using Focoos pretrained models.
🕹️ Trainer
fix missing model preprocessing when amp=True (Automatic Mixed Precision) is enabled
add COSINE scheduler quadratic warmup
add KeypointEvaluator
enhance logging with additional info
Update Visualizer (preview hook) to save RGB images instead of BGR
Restore TensorBoard Hook
📖 ModelRegistry
model registry now support automatic loading json configs from registry folder instead of declare model configs manually
🏞️ Processor
add image_size into init instead of preprocess methods
improve image loader performance
add non-blocking image transfer
optimize preprocessor speed
add focoos palette to annotators
📖 Docs
add RTMO docs
update Readme, Docs and notebook with from focoos import x for all exported classes and functions instead of absolute path

Author: andry2327

.gitignore

@@ -90,14 +90,14 @@ notebooks/.data
 .venv
 /data
 tests/junit.xml
-notebooks/datasets
-notebooks/experiments
 site/
 /datasets/
 /examples/
-notebooks/test.ipynb
+notebooks/
 gradio/output/
 tutorials/experiments
-experiments/
-notebooks/
-wandb/
+experiments
+experiments_debug
+*.pth
+.vscode
+

.vscode/settings.json

@@ -9,4 +9,20 @@
     "[python]": {
         "editor.defaultFormatter": "charliermarsh.ruff"
     },
+    "cursorpyright.analysis.autoImportCompletions": true,
+    "cursorpyright.analysis.typeCheckingMode": "basic",
+    "files.autoSave": "afterDelay",
+    "files.autoSaveDelay": 1000,
+    "editor.formatOnSave": true,
+    "[jupyter-notebook]": {
+        "files.autoSave": "off",
+        "editor.formatOnSave": false
+    },
+    "jupyter.interactiveWindow.textEditor.executeSelection": true,
+    "notebook.output.textLineLimit": 30,
+    "jupyter.askForKernelRestart": false,
+    "jupyter.alwaysTrustNotebooks": true,
+    "files.exclude": {
+        "**/.ipynb_checkpoints": true
+    }
 }

README.md

@@ -1,3 +1,7 @@
+<a href="https://www.focoos.ai" target="_blank">
+  <img src="https://public.focoos.ai/library/focoos_banner.png" alt="FocoosAI" style="max-width:100%;">
+</a>
+
 ![Tests](https://github.com/FocoosAI/focoos/actions/workflows/test.yml/badge.svg??event=push&branch=main)
 [![Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/FocoosAI/focoos/blob/main/tutorials/training.ipynb)
 [![Documentation](https://img.shields.io/badge/docs-latest-blue)](https://focoosai.github.io/focoos/)
@@ -16,7 +20,7 @@ Whether you're working in the cloud or on edge devices, the Focoos library seaml
 ### Key Features 🔑
 
 1. **Frugal Pretrained Models** 🌿
-   Get started quickly by selecting one of our efficient, [pre-trained models](https://focoosai.github.io/focoos/models/models/) that best suits your data and application needs.
+   Get started quickly by selecting one of our efficient, [pre-trained models](https://focoosai.github.io/focoos/models/) that best suits your data and application needs.
    Focoos Model Registry give access to 11 pretrained models of different size from different families: RTDetr, Maskformer, BisenetFormer
 
 2. **Fine Tune Your Model** ✨ Adapt the model to your specific use case by customize its config and training it on your own dataset.
@@ -41,11 +45,11 @@ uv pip install 'focoos @ git+https://github.com/FocoosAI/focoos.git'
 from focoos import ModelManager
 
 
-im = Image.open("image.jpg")
+im = "https://public.focoos.ai/samples/motogp.jpg" # can be local/remote path, np.array, PIL image
 
 model = ModelManager.get("fai-detr-l-obj365") # any models from ModelRegistry, FocoosHub or local folder
 
-detections = model(im)
+detections = model.infer(im,annotate=true)
 
 ```
 
@@ -110,7 +114,7 @@ Using Focoos AI helps you save both time and money while delivering high-perform
 - **4x Cheaper** 💰: Our models require up to 4x less computational power, letting you save on hardware or cloud bill while ensuring high-quality results.
 - **Tons of CO2 saved annually per model** 🌱: Our models are energy-efficient, helping you reduce your carbon footprint by using less powerful hardware with respect to mainstream models.
 
-See the list of our models in the [models](https://focoosai.github.io/focoos/models/models) section.
+See the list of our models in the [models](https://focoosai.github.io/focoos/models/) section.
 
 ---
 ### Start now!

docs/api/hub.md

@@ -1,4 +1,3 @@
-::: focoos.hub.api_client
 ::: focoos.hub.focoos_hub
 ::: focoos.hub.remote_dataset
 ::: focoos.hub.remote_model

docs/cli.md

@@ -36,6 +36,9 @@ focoos predict --model fai-detr-m-coco --source image.jpg
 
 # Export model
 focoos export --model fai-detr-m-coco --format onnx
+
+# Launch gradio interface
+focoos gradio
 ```
 
 ## 📚 Usage
@@ -47,7 +50,7 @@ focoos COMMAND [OPTIONS]
 ```
 
 Where:
-- **COMMAND**: Main operations like `train`, `val`, `predict`, `export`, `benchmark`, `hub`
+- **COMMAND**: Main operations like `train`, `val`, `predict`, `export`, `benchmark`, `gradio`, `hub`
 - **OPTIONS**: Command-specific flags and parameters with intelligent defaults
 
 ## 🛠️ Available Commands
@@ -60,6 +63,9 @@ Where:
 | **`predict`**   | Run inference on images            | `focoos predict --model fai-detr-m-coco --source image.jpg` |
 | **`export`**    | Export models to different formats | `focoos export --model fai-detr-m-coco --format onnx`       |
 | **`benchmark`** | Benchmark model performance        | `focoos benchmark --model fai-detr-m-coco --iterations 100` |
+| **`gradio`**    | Launch interactive web interface   | `focoos gradio`                                             |
+
+
 
 ### Hub Commands
 | Command            | Description                      | Example                                |
@@ -217,6 +223,23 @@ focoos hub datasets
 focoos hub datasets --include-shared
 ```
 
+### 🖥️ Interactive Web Interface
+
+```bash
+# Launch Gradio web interface
+focoos gradio
+```
+
+The Gradio interface provides an interactive web-based experience for running inference with Focoos models:
+
+- **Image Inference**: Upload images and run detection/segmentation with real-time results
+- **Video Inference**: Process video files with object detection and tracking
+- **Model Selection**: Choose from available pretrained models
+- **Confidence Tuning**: Adjust detection thresholds interactively
+- **Visual Results**: View annotated outputs with bounding boxes and masks
+
+The interface will automatically open in your default web browser, typically at `http://localhost:7860`.
+
 ## ⚙️ Configuration Options
 
 ### Common Parameters

docs/concepts.md

@@ -28,6 +28,7 @@ The Focoos Hub is a cloud-based model repository where you can store, share, and
 **Requirements**: Valid API key for private models, internet connection for initial download.
 
 ```python
+from focoos import FocoosHub, ModelManager
 # Loading from hub using hub:// protocol
 # The model is automatically downloaded and cached locally
 hub = FocoosHUB(api_key="your_api_key")
@@ -51,6 +52,7 @@ The Model Registry contains curated, pretrained models that are immediately avai
 **Requirements**: No internet connection needed, models are bundled with the library.
 
 ```python
+from focoos import ModelRegistry, ModelManager
 # Loading pretrained models from registry
 # Object detection model trained on COCO dataset
 model = ModelManager.get("fai-detr-l-coco")
@@ -59,7 +61,6 @@ model = ModelManager.get("fai-detr-l-coco")
 model = ModelManager.get("fai-mf-l-ade")
 
 # Check available models first
-from focoos import ModelRegistry
 available_models = ModelRegistry.list_models()
 print("Available models:", available_models)
 
@@ -133,7 +134,7 @@ model_info = ModelInfo(
 model = ModelManager.get("custom_detector", model_info=model_info)
 ```
 
-### Predict
+### Inference
 
 Performs end-to-end inference on input images with automatic preprocessing and postprocessing. The model accepts input images in various formats including:
 
@@ -151,7 +152,9 @@ The input images are automatically preprocessed to the correct size and format r
 This provides a simple, unified interface for running inference regardless of the underlying model architecture or task.
 
 **Parameters:**
-- `inputs`: Input images in various supported formats (`PIL.Image.Image`, `numpy.ndarray`, `torch.Tensor`)
+- `image`: Input image in various supported formats (`PIL.Image.Image`, `numpy.ndarray`, `torch.Tensor`, local or remote path)
+- `threshold`: detections threshold
+- `annotate`: if you want to annotate detections on provided image
 - `**kwargs`: Additional arguments passed to postprocessing
 
 **Returns:** [`FocoosDetections`](/focoos/api/ports/#focoos.ports.FocoosDetections) containing detection/segmentation results
@@ -161,15 +164,17 @@ This provides a simple, unified interface for running inference regardless of th
 from PIL import Image
 
 # Load an image
-image = Image.open("example.jpg")
+im_path = "example.jpg"
 
 # Run inference
-detections = model(image)
+detections = model.infer(im_path,threshold=0.5,annotate=True)
 
 # Access results
 for detection in detections.detections:
     print(f"Class: {detection.label}, Confidence: {detection.conf}")
     print(f"Bounding box: {detection.bbox}")
+
+Image.fromarray(detections.image)
 ```
 
 ### Training
@@ -259,7 +264,7 @@ infer_model = model.export(
 results = infer_model(input_image)
 ```
 
-### Predict
+### Inference
 
 Performs end-to-end inference on input images with automatic preprocessing and postprocessing on the selected runtime. The model accepts input images in various formats including:
 
@@ -277,7 +282,9 @@ The input images are automatically preprocessed to the correct size and format r
 This provides a simple, unified interface for running inference regardless of the underlying model architecture or task.
 
 **Parameters:**
-- `inputs`: Input images in various supported formats (`PIL.Image.Image`, `numpy.ndarray`, `torch.Tensor`)
+- `image`: Input image in various supported formats (`PIL.Image.Image`, `numpy.ndarray`, `torch.Tensor`, local or remote path)
+- `threshold`: detections threshold
+- `annotate`: if you want to annotate detections on provided image
 - `**kwargs`: Additional arguments passed to postprocessing
 
 **Returns:** [`FocoosDetections`](/focoos/api/ports/#focoos.ports.FocoosDetections) containing detection/segmentation results
@@ -287,14 +294,14 @@ This provides a simple, unified interface for running inference regardless of th
 from PIL import Image
 
 # Load an image
-image = Image.open("example.jpg")
+image_path = "example.jpg"
 
 # Run inference
 infer_model = model.export(
     runtime_type=RuntimeType.TORCHSCRIPT_32,
     out_dir="./exported_models"
 )
-detections = infer_model(image)
+detections = infer_model.infer(image_path,threshold=0.5, annotate = True)
 
 # Access results
 for detection in detections.detections:

docs/hub/overview.md docs/hub/index.mddocs/hub/overview.md renamed to docs/hub/index.md

@@ -44,7 +44,7 @@ Remote models can also be called directly like functions:
 
 ```python
 # This is equivalent to calling remote_model.infer()
-results = remote_model("path/to/image.jpg", threshold=0.5)
+results = remote_model.infer("path/to/image.jpg", threshold=0.5)
 ```
 
 ## Supported Input Types
@@ -129,13 +129,10 @@ for i, detection in enumerate(results.detections):
 Visualize results using the built-in utilities:
 
 ```python
-from focoos import annotate_image
 
-results = model.infer(image=image, threshold=0.5)
+results = model.infer(image=image, threshold=0.5,annotate=True)
 
-annotated_image = annotate_image(
-    im=image, detections=results, task=model.model_info.task, classes=model.model_info.classes
-)
+Image.fromarray(results.image)
 ```
 
 ## Model Management for Remote Inference

docs/hub/remote_inference.md

@@ -56,7 +56,7 @@ Using the model is as simple as it could! Just call it with an image.
 ```python
 from PIL import Image
 image = Image.open("<PATH-TO-IMAGE>")
-detections = model(image)
+detections = model.infer(image)
 ```
 
 `detections` is a [FocoosDetections](/focoos/api/ports/#focoos.ports.FocoosDetections) object, containing a list of [FocoosDet](/focoos/api/ports/#focoos.ports.FocoosDet) objects and optionally a dict of information about the latency of the inference. The `FocoosDet` object contains the following attributes:
@@ -66,13 +66,14 @@ detections = model(image)
 - `cls_id`: Class ID (0-indexed).
 - `label`: Label (name of the class).
 - `mask`: Mask (base64 encoded string having origin in the top left corner of bbox and the same width and height of the bbox).
+- `keypoints`: keypoints detected
 
-If you want to visualize the result on the image, there's a utily for you.
+If you want to visualize the result on the image,just set annotate=true
 
 ```python
-from focoos import annotate_image
-
-annotate_image(image, detections, task=model.model_info.task, classes=model.model_info.classes).save("predictions.png")
+from PIL import Image
+detections = model.infer(image,annotate=True)
+Image.fromarray(detections.image)
 ```
 
 ## 2. 🔥 PyTorch Inference
@@ -118,9 +119,9 @@ Now, again, you can now run the model by simply passing it an image and visualiz
 ```python
 from focoos import annotate_image
 
-detections = model(image)
+detections = model.infer(image,annotate=True)
 
-annotate_image(image, detections, task=model.model_info.task, classes=model.model_info.classes).save("predictions.png")
+Image.fromarray(detections.image)
 ```
 
 `detections` is a [FocoosDetections](/focoos/api/ports/#focoos.ports.FocoosDetections) object.
@@ -158,15 +159,16 @@ Let's visualize the output. As you will see, there are not differences from the
 ```python
 from focoos import annotate_image
 
-detections = optimized_model(image)
-annotate_image(image, detections, task=model.model_info.task, classes=model.model_info.classes).save("prediction.png")
+detections = optimized_model(image, annotate = True)
+Image.fromarray(detections.image)
 ```
+
 `detections` is a [FocoosDetections](/focoos/api/ports/#focoos.ports.FocoosDetections) object.
 
 
 But, let's see its latency, that should be substantially lower than the pure pytorch model.
 ```python
-optimized_model.benchmark(iterations=10, size=512)
+optimized_model.benchmark(iterations=10)
 ```
 
 You can use different runtimes that may fit better your device, such as TensorRT. See the list of available Runtimes at [`RuntimeTypes`](/focoos/api/ports/#focoos.ports.RuntimeType). Please note that you need to install the relative packages for onnx and tensorRT for using them.

docs/inference.md

@@ -39,6 +39,19 @@ With the Focoos SDK, you can take advantage of a collection of foundational mode
 | [fai-mf-m-coco-ins](fai_mf.md) | [Mask2Former](https://github.com/facebookresearch/Mask2Former) ([Resnet-101](https://github.com/pytorch/vision/blob/main/torchvision/models/resnet.py)) | Common Objects (80) | [COCO](https://cocodataset.org/#home) | segm/AP: 43.09<br>segm/AP50: 65.87 | 70 |
 | [fai-mf-l-coco-ins](fai_mf.md) | [Mask2Former](https://github.com/facebookresearch/Mask2Former) ([Resnet-101](https://github.com/pytorch/vision/blob/main/torchvision/models/resnet.py)) | Common Objects (80) | [COCO](https://cocodataset.org/#home) | segm/AP: 44.23<br>segm/AP50: 67.53 | 55 |
 
+<small> AP = Average Precision averaged by class </small> <br>
+<small> AP50 = Average Precision at IoU threshold 0.50 averaged by class </small> <br>
+<small> FPS = Frames per second computed using TensorRT with resolution 640x640 </small> <br>
+
+## Keypoint Detection 🥷
+
+| Model Name | Architecture | Domain (Classes) | Dataset | Metric | FPS Nvidia-T4 |
+|------------|--------------|------------------|----------|---------|--------------|
+| [rtmo-s-coco](rtmo.md) | [RTMO](https://github.com/open-mmlab/mmpose/tree/main/projects/rtmo) ([CSP-Darknet](https://github.com/open-mmlab/mmpose/blob/main/mmpose/models/backbones/csp_darknet.py)) | Persons (1) | [COCO](https://cocodataset.org/#home) | keypoints/AP: 67.94<br>keypoints/AP50: 87.86 | 104 |
+| [rtmo-m-coco](rtmo.md) | [RTMO](https://github.com/open-mmlab/mmpose/tree/main/projects/rtmo) ([CSP-Darknet](https://github.com/open-mmlab/mmpose/blob/main/mmpose/models/backbones/csp_darknet.py)) | Persons (1) | [COCO](https://cocodataset.org/#home) | keypoints/AP: 70.94<br>keypoints/AP50: 89.47 | 89 |
+| [rtmo-l-coco](rtmo.md) | [RTMO](https://github.com/open-mmlab/mmpose/tree/main/projects/rtmo) ([CSP-Darknet](https://github.com/open-mmlab/mmpose/blob/main/mmpose/models/backbones/csp_darknet.py)) | Persons (1) | [COCO](https://cocodataset.org/#home) | keypoints/AP: 72.14<br>keypoints/AP50: 89.85 |  63 |
+
+
 <small> AP = Average Precision averaged by class </small> <br>
 <small> AP50 = Average Precision at IoU threshold 0.50 averaged by class </small> <br>
 <small> FPS = Frames per second computed using TensorRT with resolution 640x640 </small> <br>