Android build failed when using OpenGL ES3 Graphics API

[trackme518]

Hi,
I am trying to make it work on Android. I verified that the decoder and encoder models are not null (ie loaded). I can pick the image from gallery and load it as a texture to Raw Image but when I call segmentation, nothing happens - no error and no mask are created. To be sure that it is not a problem with texture I have also tried to load the texture, get the pixels and set them to new texture with TextureFormat.RGB24 format (as mobileSAM is expecting RGB not A channel). That did not help either.

Does it work on Android? Has anyone else tried?

    public async void SetImageTexture(Texture2D texture){
      // Create a new RGB Texture2D
      Texture2D rgbTexture = new Texture2D(texture.width, texture.height, TextureFormat.RGB24, false, true, true);

      // Read the pixels from the source texture and copy them to the new RGB texture, ignoring the alpha channel
      Color[] rgbaColors = texture.GetPixels();
      Color[] rgbColors = new Color[rgbaColors.Length];

      for (int i = 0; i < rgbaColors.Length; i++){
          rgbColors[i] = new Color(rgbaColors[i].r, rgbaColors[i].g, rgbaColors[i].b );
      }

      rgbTexture.SetPixels(rgbColors);
      rgbTexture.Apply();
      _texture = rgbTexture;

      rawImage.texture = _texture;
      _mobileSAMPredictor.SetImage(_texture);
      // Example prompt: a single point in the middle of the image    
      float[] pointCoords = new float[] { _texture.width / 2f, _texture.height / 2f };
      float[] pointLabels = new float[] { 1f };  // 1 for foreground point
      // Perform segmentation
      _mobileSAMPredictor.Predict(pointCoords, pointLabels);
      // Retrieve the result
      segmentResult = _mobileSAMPredictor.Result;//segmentResult is RenderTexture
      MaskImage.texture = _mobileSAMPredictor.Result;
      }

Unity 6000.0.29f1
Sentis 2.1
build target Android
Device: Google Pixel 8a, Android 15
OS Linux

[trackme518]

I saved the result image into Texture2D and to the phone's gallery - it exists alright but it is completely empty (ie 100% alpha).

[julienkay]

Hmm, I did a quick test on Android with the sample scene from the package and it seems to work fine for me.
I tested with Unity 2022.3 LTS, Vulkan, on a phone with Android 13.
Maybe you can do a test with the sample scene as well to see if it's specific to the device you're testing with?
You could also try to set the Backend to CPU for testing. Will be a lot slower, but maybe it's an issue with the GPU backend on your specific device.

_mobileSAMPredictor = new MobileSAM();
_mobileSAMPredictor.Backend = Unity.Sentis.BackendType.CPU

[trackme518]

Thank a lot - interestingly in your source code you are already setting the backend to CPU: private BackendType _backend = BackendType.CPU; in MobileSAM.cs line 69 (https://github.com/julienkay/com.doji.mobilesam/blob/master/com.doji.mobilesam/Runtime/Scripts/MobileSAM.cs). I am using Pixel 8a, Android 15, Unity Sentis 2.1. I will test with the sample as well.

[julienkay]

Oh, that's a good point, I should probably change that. I usually default to GPU, I think I left it as CPU here by mistake.
Both backends should work though, but If the CPU backend is not even working there must be something else going on.

[trackme518]

I was having hard time making the sample work as it was in PackageCache (added through package manager) and marked invisible - even if I mark it visible it is in immutable location so Unity can not create meta file for the folder. So, instead I have removed the package and manually added the package into Assets folder, then added Sentis 2.1 again (as since I added the package manually it will not solve dependency for me).

Also I was using new input system which is not compatible with the old one, so I changed my Player settings to enable both instead. I have unloaded my scene and activated the sample scene. I got timeout when reloading the project but it initiated in the end. I tried to run it in Editor (Linux, Ubuntu 24.04 LTS). I got an error MissingReferenceException: The variable m_Texture of RawImage doesn't exist anymore. I assigned the sample texture athos to RawImage component in Editor. When I play the scene it works as expected.

Finally, I started to build for Android on device. I got error about Android not supporting both input systems (old and new), I choose to ignore it and try to build anyway. That did not work so I got back to Player settings and set the input system to old only. When the scene launch on mobile it will let me click and green circle will appear but no selection/mask is created / showed.

So even the sample scene seems not working properly on Android for me.

Just to be sure I set in my Player setting input system to old only and tried to rebuild for android. But I don't think it has anything to do with it as the markers were placed correctly even when I had "both" for input system. Anyway the result is the same - I can place markers but nothing happens beside that. Also the sample project is clearly intended for desktop as there is not right click button for mobile but it does not matter for testing purposes.

[trackme518]

The model get loaded even on Android as per LogCat logs:

2025/02/02 16:57:25.877 2055 2088 Info Unity Doji.AI.Segmentation.MobileSAM:get_Decoder()
2025/02/02 16:57:25.877 2055 2088 Info Unity Doji.AI.Segmentation.MobileSAM:InitializeNetwork()
2025/02/02 16:57:25.877 2055 2088 Info Unity Doji.AI.Segmentation.Samples.MobileSAM_BasicSample:Start()
2025/02/02 16:57:25.877 2055 2088 Info Unity 
2025/02/02 16:57:25.877 2055 2088 Info Unity MobileSAM encoder ONNX model LOADED. (I added this myself - when model is not null)
2025/02/02 16:57:25.877 2055 2088 Info Unity UnityEngine.DebugLogHandler:Internal_Log(LogType, LogOption, String, Object)
2025/02/02 16:57:25.877 2055 2088 Info Unity Doji.AI.Segmentation.MobileSAM:get_Decoder()
2025/02/02 16:57:25.877 2055 2088 Info Unity Doji.AI.Segmentation.MobileSAM:InitializeNetwork()
2025/02/02 16:57:25.877 2055 2088 Info Unity Doji.AI.Segmentation.Samples.MobileSAM_BasicSample:Start()

But no masks are generated.

[trackme518]

Can you send me the whole Unity project in zip file so I can test that please?

I need to see if it has something to do with project settings or not.

I have sent you my email on your LinkedIn.

[julienkay]

As a side note, if you've added the package through the Package Manager, you should be able to import the samples there as well:

But it seems you got it working correctly in-Editor and it still didn't work on-device, so it increasingly looks like a device-specific issue. Hard to be sure / troubleshoot if there are no errors though. At this point I'd suggest posting over on the Unity forum in the Sentis category, to get the chance of someone from Unity seeing it.

I'll send the project here. I doubt it will change much tbh, but can't hurt to try.
MobileSAM_SampleProject.zip

[trackme518]

I will try to install your version of Unity as well and test with that. Thanks for the samples import tip - I didn't know it's an option.

[trackme518]

:-D It is working on the older Unity version LOL :-D. I guess, now I can try to rewrite my whole project to the older Unity. At least now we know there is something causing problems in Unity 6 LTS with this package. @julienkay maybe you should update the info on Asset store to note this (by default it is assumed the package works with latest LTS).

[julienkay]

Good to hear, that it's working!

Though the strange thing is, that I've also tested the project that I sent you with Unity 6000.0.31f1 and it worked for me as well. Maybe it was some project settings after all?

[trackme518]

Very interesting - I am actually having problem with migration to lower version as there are many inter dependencies so it would be nice to make it work.

I was using 6000.0.029f1 originally

When I was using your project I needed to switch to ARM64 for the mobile and from MONO to IL2CPP not sure if that is the factor or not. Sadly I am not sure what might be the cause right now.

[trackme518]

I tested the original scene with 6000.0.029f1 as a new project - it works. So i suspect it has to do with some other packages. It takes a long time as I need to build the project between changes.

[trackme518]

Ok I have a cause:
Added packages:
"com.unity.cloud.gltfast": "6.8.0",
"com.unity.xr.arfoundation": "6.0.5",
"com.unity.xr.interaction.toolkit": "3.0.7",
Imported interaction.toolkit starter assets

This breaks the functionality:
switched graphics API to OpenGL ES3

The problem is that it is needed for ARCore :-(

Is this the problem with specific model used (ie MobileSAM)? Or is this a general problem regarding Sentis? Would be possible to somehow enable OpenGL for the model?