Bug: GPU Delegate failure causes predictor to be null, inference completely fails without CPU fallback

[dmjtian]

Search before asking

• I have searched https://github.com/ultralytics/yolo-flutter-app/issues and did not find a similar report.

Project area

Plugin API

Bug

Bug Description

When TensorFlow Lite GPU Delegate fails at runtime (e.g., out of memory, unsupported operations), the Interpreter constructor throws an exception that is not caught. This causes the predictor to remain null, and all inference is skipped silently.

This issue was discovered while debugging Issue #413 - the CLASSIFY task showed FPS: 0.0 and 0 detections because the GPU Delegate failed to initialize.

Error Logs

E/YOLOView: java.lang.IllegalArgumentException: Internal error: Failed to apply delegate: Failed to build program executable - Out of host memory
E/YOLOView: TfLiteGpuDelegate Prepare: delegate is not initialized
E/YOLOView: Node number 517 (TfLiteGpuDelegateV2) failed to prepare.
W/YOLOView: predictor is null, skipping inference

Root Cause

In all Predictor classes (Classifier.kt, ObjectDetector.kt, Segmenter.kt, PoseEstimator.kt, ObbDetector.kt), the GPU Delegate is added in interpreterOptions, but if the Interpreter constructor fails at runtime, there's no try-catch to fall back to CPU mode:

// Current code - no fallback
interpreter = Interpreter(modelBuffer, interpreterOptions)  // Throws exception if GPU fails

Expected Behavior

When GPU Delegate fails, the code should automatically fall back to CPU inference instead of leaving predictor as null.

Suggested Fix

Wrap Interpreter creation in try-catch and retry with CPU-only options:

interpreter = try {
    Interpreter(modelBuffer, interpreterOptions)
} catch (e: Exception) {
    if (gpuDelegateRequested) {
        Log.w(TAG, "GPU delegate failed at runtime, falling back to CPU: ${e.message}")
        gpuDelegate?.close()
        val cpuOptions = Interpreter.Options().apply { setNumThreads(4) }
        Interpreter(modelBuffer, cpuOptions)
    } else {
        throw e
    }
}

Affected Files

- android/src/main/kotlin/com/ultralytics/yolo/Classifier.kt
- android/src/main/kotlin/com/ultralytics/yolo/ObjectDetector.kt
- android/src/main/kotlin/com/ultralytics/yolo/Segmenter.kt
- android/src/main/kotlin/com/ultralytics/yolo/PoseEstimator.kt
- android/src/main/kotlin/com/ultralytics/yolo/ObbDetector.kt

Related Issues

- #413 (CLASSIFY task not outputting results)


### Environment

### Environment

- Device: Redmi K60
- Flutter:  3.38.5
- Plugin version: latest main branch


### Minimal Reproducible Example

_No response_

### Additional

_No response_

### Are you willing to submit a PR?

- [x] Yes I'd like to help by submitting a PR!

[dmjtian]

#419 #413
As previously stated, the classification model may not produce outputs due to insufficient GPU memory, which would result in the model falling back to the CPU. The following two images are presented: the first illustrates the effect without output results, while the second demonstrates the modified effect.

[UltralyticsAssistant]

👋 Hello @dmjtian, thank you for submitting a ultralytics/yolo-flutter-app 🚀 Issue. This is an automated response to help us triage quickly—an Ultralytics engineer will also assist soon 🛠️🙂

To help us address your GPU Delegate failure + CPU fallback concern efficiently, please ensure you've provided the following information:

1. For bug reports:

• A clear and concise description of the bug
• A minimum reproducible example MRE that demonstrates the issue (currently missing—please add one 🙏)
• Your environment details (OS, Python version, package versions)
• Expected behavior vs. actual behavior
• Any error messages or logs related to the issue

2. For feature requests:

• A clear and concise description of the proposed feature
• The problem this feature would solve
• Any alternative solutions you've considered

3. For questions:

• Provide as much context as possible about your question
• Include any research you've already done on the topic
• Specify which parts of the documentation, if any, you've already consulted

Additional details that would be especially helpful for this report 🔎:

• Exact Android OS version on the Redmi K60
• Whether the failure reproduces consistently or only under memory pressure / certain models
• Full logs around interpreter/delegate initialization (the snippet you shared is great—more context helps)
• Confirmation of the exact commit on main you tested (since you mentioned “latest main branch”)

Please make sure you've searched existing issues to avoid duplicates (thanks for doing that ✅). If you need to add any additional information, please comment on this issue.

[dmjtian]

GPU Fallback Issue 🔧

You're right that GPU delegate fallback shouldn't make classification "empty". However, I discovered that the current implementation doesn't have a CPU fallback mechanism when GPU delegate fails at runtime (e.g., out of memory).

I've opened a separate issue #419 to track this GPU fallback problem, as it affects all predictor classes (Classifier, ObjectDetector, Segmenter, PoseEstimator, ObbDetector), not just classification.

This PR (#418) focuses on fixing the channel-mapping issue for classification results. Let me know if you'd like me to clarify anything else!

[pderrenger]

Thanks for separating concerns—keeping #418 focused on classification channel mapping and tracking the GPU-delegate runtime failure/CPU fallback here in #419 makes sense; if you can add a minimal repro to this issue and confirm it still happens on the latest published plugin (not just main), we’d welcome a PR that wraps Interpreter(...) creation with a GPU→CPU retry (closing the failed delegate) and surfaces a clear error instead of leaving the predictor null.

[dmjtian]

After further testing, I found that the "no output" issue was actually caused by #413
(classification results not being passed to Flutter UI), not GPU memory failure.

With the #413 fix applied, classification works correctly on my device (Redmi K60,
HyperOS 2.0, Android 15) with GPU delegate enabled.

I do have a defensive GPU→CPU fallback implementation ready, but since I cannot
reproduce the GPU failure scenario, I'll hold off on submitting that PR unless
others encounter the issue.

[pderrenger]

Thanks for the follow-up—glad the “no output” behavior was resolved by the #413 UI plumbing fix and that GPU delegate is stable for you on Redmi K60 (HyperOS 2.0 / Android 15); even without a local repro, a small PR that defensively wraps Interpreter(...) init with a GPU→CPU retry (closing the failed delegate) and returns a clear error to Flutter if both paths fail would still be a net win, so if you decide to open it please base it on the latest published plugin release and include whatever logs/steps you have for exercising the fallback path.

[asabri97]

@dmjtian I've tried reproducing this on Android - I have Pixel 8a. I am not able to do so. It seems like the issue occurs on low GPU phone but that I believe is expected.