fix: Add classification streaming support for YOLOView

[dmjtian]

Summary

Adds classification result processing to convertResultToStreamData() method in Android's YOLOView.kt, enabling real-time classification streaming support.

Problem

Currently, classification task works for single image prediction (via YOLO.predict()) thanks to PR #418, but real-time streaming (via YOLOView with onResult callback) returns empty results.

Current Behavior

YOLOView(
  task: YOLOTask.classify,
  onResult: (results) => print(results.length), // Always prints 0
)

Logs:

D/Classifier: Classification result: top1Label=corn, conf=0.84, index=0
D/Classifier: Prediction completed successfully
D/YOLOView: ✅ Total detections in stream: 0 (boxes: 0, obb: 0)

Classifier succeeds but results array is empty because convertResultToStreamData() doesn't process result.probs.

Root Cause

The convertResultToStreamData() method in android/src/main/kotlin/com/ultralytics/yolo/YOLOView.kt handles:

• ✅ Detection boxes (result.boxes)
• ✅ Pose keypoints (result.keypointsList)
• ✅ OBB (result.obb)
• ❌ Classification results (result.probs) - MISSING

Solution

Add classification result processing in convertResultToStreamData() method, similar to how other task types are handled.

Code Changes

File: android/src/main/kotlin/com/ultralytics/yolo/YOLOView.kt

Location: After line ~1668 (after the includeDetections block ends)

Add this code block:

// Add classification results (if available and enabled for CLASSIFY task)
if (config.includeClassifications && result.probs != null) {
    val probs = result.probs!!
    Log.d(TAG, "🎯 Processing CLASSIFY result: top1Label=${probs.top1Label}, conf=${probs.top1Conf}, index=${probs.top1Index}")

    // Add classification result to detections array (for compatibility with YOLOResult.fromMap)
    val detections = map["detections"] as? ArrayList<Map<String, Any>> ?: ArrayList()

    val classificationDetection = HashMap<String, Any>()
    classificationDetection["classIndex"] = probs.top1Index
    classificationDetection["className"] = probs.top1Label
    classificationDetection["confidence"] = probs.top1Conf.toDouble()

    // Full image bounding box for classification
    val boundingBox = HashMap<String, Any>()
    boundingBox["left"] = 0.0
    boundingBox["top"] = 0.0
    boundingBox["right"] = result.origShape.width.toDouble()
    boundingBox["bottom"] = result.origShape.height.toDouble()
    classificationDetection["boundingBox"] = boundingBox

    // Normalized bounding box (full image)
    val normalizedBox = HashMap<String, Any>()
    normalizedBox["left"] = 0.0
    normalizedBox["top"] = 0.0
    normalizedBox["right"] = 1.0
    normalizedBox["bottom"] = 1.0
    classificationDetection["normalizedBox"] = normalizedBox

    detections.add(classificationDetection)
    map["detections"] = detections

    Log.d(TAG, "✅ Added classification result: ${probs.top1Label} (conf=${probs.top1Conf}, index=${probs.top1Index})")
}

Optional but recommended: Update the log message to include probs count:

Before:

Log.d(TAG, "✅ Total detections in stream: ${detections.size} (boxes: ${result.boxes.size}, obb: ${result.obb.size})")

After:

Log.d(TAG, "✅ Total detections in stream: ${detections.size} (boxes: ${result.boxes.size}, obb: ${result.obb.size}, probs: ${if (result.probs != null) 1 else 0})")

Testing

Test Environment

• Model: yolo11n-cls.tflite
• Task: YOLOTask.classify
• Platform: Android (Pixel 6, Android 14)
• Flutter: 3.x

Before Fix

D/YOLOView: ✅ Total detections in stream: 0
I/flutter: Results: 0

After Fix

D/YOLOView: 🎯 Processing CLASSIFY result: top1Label=corn, conf=0.84, index=0
D/YOLOView: ✅ Added classification result: corn (conf=0.84, index=0)
D/YOLOView: ✅ Total detections in stream: 1 (boxes: 0, obb: 0, probs: 1)
I/flutter: Results: 1
I/flutter:   className: corn
I/flutter:   confidence: 0.84

Test Code

YOLOView(
  modelPath: 'assets/yolo11n-cls.tflite',
  task: YOLOTask.classify,
  confidenceThreshold: 0.3,
  onResult: (results) {
    print('Results: ${results.length}');
    if (results.isNotEmpty) {
      print('  className: ${results[0].className}');
      print('  confidence: ${results[0].confidence}');
    }
  },
)

Design Decisions

Why add to detections array?

• Maintains consistency with other task types (detection, pose, OBB)
• Compatible with existing YOLOResult.fromMap() deserialization
• Allows Flutter layer to handle all results uniformly

Why use full image bounding box?

• Classification applies to the entire image, not a specific region
• Provides consistent data structure with other task types
• Simplifies Flutter layer processing

Why check config.includeClassifications?

• Respects YOLOStreamingConfig settings
• Allows users to disable classification results if needed
• Follows the same pattern as other task types

Impact

Fixed

• ✅ Real-time classification streaming now works
• ✅ YOLOView with onResult returns classification results
• ✅ Consistent behavior across all task types

Not Affected

• Single image classification (YOLO.predict()) - already works (PR Fix Android CLASSIFY task: ensure FloatArray compatibility (Issue #413 follow-up) #418)
• Detection, segmentation, pose, OBB tasks - unchanged
• iOS implementation - may need similar fix (separate PR)

Related

• Related to: PR Fix Android CLASSIFY task: ensure FloatArray compatibility (Issue #413 follow-up) #418 (fixed single image classification)
• Complements: v0.1.43 release
• Platform: Android (iOS similar fix)

Checklist

• Code compiles without errors
• Tested on physical Android device
• Follows existing code style and patterns
• Includes appropriate logging
• No breaking changes

Notes

This PR focuses on Android implementation. iOS may need a similar fix in ios/Classes/YOLOView.swift if the issue exists there.

---

Author: @dmjtian (original contributor of PR #418)

## 🛠️ PR Summary

<sub>Made with ❤️ by [Ultralytics Actions](https://www.ultralytics.com/actions)</sub>

### 🌟 Summary
Adds classification streaming support in Android `YOLOPlugin` by aligning emitted results with `Results.summary()` for consistent consumption in `YOLOView` 📡

### 📊 Key Changes
-Standardized `response["classification"]` payload to `{name, class, confidence}` to match `Results.summary()` format
-Added `top5` as a structured list of `{name, class, confidence}` entries for streaming consumers
-Retained backward compatibility by still populating `response["boxes"]` with a full-image “box” containing top-1 classification data

### 🎯 Purpose & Impact
-Enables reliable classification streaming in `YOLOView` without requiring hardcoded label mappings ✅
-Improves downstream parsing consistency across platforms/features by mirroring `Results.summary()` output 🧩
-Reduces risk of breaking existing apps by keeping legacy `boxes`-based compatibility behavior intact 🔁

[UltralyticsAssistant]

👋 Hello @dmjtian, thank you for submitting a ultralytics/yolo-flutter-app 🚀 PR! This is an automated message, and an Ultralytics engineer will assist shortly 🛠️

-✅ Define a Purpose: Clearly explain the purpose of your fix or feature in your PR description, and link to any relevant issues. Ensure your commit messages are clear, concise, and adhere to the project's conventions.
-✅ Synchronize with Source: Confirm your PR is synchronized with the ultralytics/yolo-flutter-app main branch. If it's behind, update it by clicking the 'Update branch' button or by running git pull and git merge main locally.
-✅ Ensure CI Checks Pass: Verify all Ultralytics Continuous Integration (CI) checks are passing. If any checks fail, please address the issues.
-✅ Update Documentation: Update the relevant documentation for any new or modified features.
-✅ Add Tests: If applicable, include or update tests to cover your changes, and confirm that all tests are passing.
-✅ Sign the CLA: Please ensure you have signed our Contributor License Agreement if this is your first Ultralytics PR by writing "I have read the CLA Document and I sign the CLA" in a new message.
-✅ Minimize Changes: Limit your changes to the minimum necessary for your bug fix or feature addition. "It is not daily increase but daily decrease, hack away the unessential. The closer to the source, the less wastage there is." — Bruce Lee

For more guidance, please refer to our Contributing Guide. Don't hesitate to leave a comment if you have any questions. Thank you for contributing to Ultralytics! 🚀

[UltralyticsAssistant]

🔍 PR Review

_{Made with ❤️ by Ultralytics Actions}

Good direction aligning classification output with Results.summary(), but there’s a correctness issue in top5 class indices and a likely backward-compatibility break in the legacy boxes payload. Fixing those should make this change safe for existing consumers.

💬 Posted 2 inline comments

[UltralyticsAssistant]

⚠️ HIGH: top5 uses mapIndexed and sets class to the enumeration index, not the model’s class index. That makes the payload incorrect if top-5 class IDs are not 0..4. This will break consumers expecting actual class indices. Use the model-provided top-5 indices instead of idx if available (e.g., probs.top5Indices).

[UltralyticsAssistant]

💡 MEDIUM: Backward compatibility is likely broken: the previous payload included className and classIndex keys inside boxes, but the new payload replaces them with name and class. Existing apps consuming the legacy fields will fail. Consider keeping the old keys in addition to the new ones for the compatibility path.

[pderrenger]

Thanks @dmjtian—this looks like the right fix for streaming; can you make sure the emitted classification payload mirrors the core Results.summary() contract (top-5 entries shaped like {name, class, confidence} derived from Probs.top5/top5conf) so downstream parsing stays consistent, and please confirm it still reproduces/fixes on the latest main before we merge (refs: Results.summary(), Probs).