Optimizations

[vmobilis]

@rmatif, hello and thank you for the program! Just a wish: when you'll make final release, could you please try to build CPU-only version in 3 flavors?

• Cortex-A53 (ARMv8-A), for older phones;
• Cortex-A55 (ARMv8.2-A);
• Cortex-A510 (ARMv9), for Snapdragon 8 Gen 1+.

A53 => A55 does speed things up in Termux, just set CMAKE_CXX_FLAGS="-mcpu=cortex-a55".

ARMv9 should also be faster, though I didn't test it.

And again, thank you!

[rmatif]

@vmobilis Thanks for the feedback and the suggestion. I have now included the variants you mentioned in the latest release, and I will make sure to include them in all future releases. I experienced a nice 33% speedup. Please let me know if it works better for you, and feel free to share any other suggestions or report any issues.

[vmobilis]

@rmatif, thank you very much!
1st impression:

• CPU works great!
• OpenCL works.
• Vulkan works (Mali GPU).

I'm sure I'm missing something 😅, but where is the SAVE IMAGE button?

Suggestions:

1. Keep prompts when changing model.

2. Display small names of currently loaded models under LOAD buttons (for model, vae, clip etc.).

3. Add threads field. In CPU mode, the phone gets too hot. There is a workaround in using OpenCL (it seems to always use single thread), but if it will not work for someone, limiting number of threads will cool CPU down.

4. Add "keep screen on" setting. Android 13+ tends to stop the application if phone goes to sleep, and wakelock does not help (or I'm doing something wrong).
   In Java it can be done as:

import android.view.Window;
import android.view.WindowManager;

Window win = getWindow();
if (win != nul) {
    win.addFlags(WindowManager.LayoutParams.KEEP_SCREEN_ON);
    //win.clearFlags(WindowManager.LayoutParams.KEEP_SCREEN_ON);
}

5. Add night mode, it is useful for OLEDs. In Java it can be done as:

import androidx.appcompat.app.AppCompatDelegate;

private static int daynightmode = AppCompatDelegate.MODE_NIGHT_YES;   // MODE_NIGHT_NO

// changing night mode will restart the activity,
// therefore set it in the 1st place
if(daynightmode != AppCompatDelegate.getDefaultNightMode()) {
    AppCompatDelegate.setDefaultNightMode(daynightmode);
}

6. Add fast preview while sampling.
   The preview technique by @stduhpf (fast latent image preview leejet/stable-diffusion.cpp#454) at 1/8 resolution is very fast, and after 8x nearest neighbor upscaling (like here, for example) it can be easily viewed as usual, not slowing sampling down.

7. Add timing for decoding step (currently decoding time is not displayed, it just looks like last sampling step takes twice longer).

8. Thank you!

[vmobilis]

@rmatif, if I'll figure out how to use Flutter, I'll try to suggest things as commits. But you'll probably implement them yourself by that time. 🙂

[rmatif]

@vmobilis Thanks for your testing and valuable suggestions.

Can you tell me a bit more about performance across the different backends please? I'm really curious, like how many sec/it for an SD1.5 at a given cfg and resolution.

For the "Save Image" button, it's not there simply because the image is saved automatically after each generation.

I will take into account all your suggestions about UI. I had already in mind some of them and will be pretty easy to implement.

Regarding the decoding time, as I'm using the callbacks to get this info, the decoding time will display at the very end of the generation process right before returning the image, so displaying it will not make much sense as it will be on the screen for less than one second. I think just displaying that the process is in the decoding stage is enough to avoid the feeling that it's stuck on the last step. Enabling VAE tiling already displays the decoding time per step. And if one is curious about all of this, there is a log button that saves all the callbacks and returns them.

I'll take a look at the PR. It will be nice to have it especially if it doesn't slow the sampling process. Will definitely test it.

For now I think I will focus a bit on the backend side to speed things up. Will be looking at ik_llama.cpp as they claim some nice speeds on ARM Neon CPUs compared to the original llama.cpp, as well as SealAILab implementation of Winograd conv as replacement of im2col+gemm, it may speed things up a bit. And try to tune the Vulkan backend a bit, which I already did a bit by disabling large matmul. Will see if I can get any further with this, which I doubt because I think some of the shaders need to be rewritten and simplified for the phone GPU, and I have no expertise on that.

You're most welcome if you want to contribute, even by simply suggesting ideas.

[vmobilis]

@rmatif, I found images, thank you! Maybe print their names with full path on saving?
CPU time is ~55 sec./it., Vulkan is ~239 sec./it.
I'll measure more precisely and report.

[vmobilis]

@rmatif, the results (Cortex-A55 build):

CPU: 60.35 s/it, the phone gets hot, small ui lags.
OpenCL: 86.12 s/it, warm, smooth ui work.
Vulkan: 245.79 s/it, cold, ui has noticeable freezes.

CPU is Helio G99.
Model is Dreamshaper 8 LCM, 5 steps, cfg-scale 1.5, sampler LCM, Karras scheduler, without flash-attention, 512x512.

[rmatif]

@vmobilis Thank you so much for the results, I may compile them if I have enough and put them somewhere in the documentation.

If you are on Mali GPU, OpenCL is doing nothing and falling back into CPU. You can monitor the usage and you'll see that it's CPU.
See here OPENCL.md. Only compatible with Adreno 7xx/8xx.

I'm currently working on tuning Vulkan, and I have achieved a 2x speedup compared to the current release, at least on my Mali G68 MC4. It's almost competitive with CPU now.

I will join the topic. I tested the application on my device with the Mediatek Dimensity 8300 Ultra CPU (Cortex-510 CPU assembly). When generating the CPU, in a resolution of 512x512 in 5 steps (absolute reality model using Taesd) - a total of 211.57s. For each step 28-32 s / it came out. Generates quickly. When using the Vulcan GPU - generates very slowly, for only one step 240 s / it came out. The interface when using the GPU slows down and lags a lot. The GPU in my device is Mali-G615 MC6.

[vmobilis]

@rmatif, another test, Redme Note 13, Snapdragon 685, Adreno 610, build for Cortex-A53.

CPU: 137.12 s/it, warm, ui is smooth.
OpenCL: 134.79 s/it, warm, smooth ui.
Vulkan: crashes when loading model.