use time step embedding from file by RSMNYS · Pull Request #928 · mlcommons/mobile_app_open

RSMNYS · 2024-10-28T20:17:23Z

Timestamp-embedding-parser

* Add TaskConfig.CustomConfig and pass them to backend * Add CustomConfig for main.cc * Use seed and num_steps from CustomConfig for TFLite backend * Replace std::cout with LOG(INFO) * Format files

* Add ConvertOutputs() API * Add ConvertOutputs() for mobile_back_tflite * Set minimum macos version * Set minimum macos version to 13.1 * Update _kIphoneOnGitHubAction

github-actions · 2024-10-28T20:17:34Z

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

sonarqubecloud · 2024-10-28T20:58:35Z

Quality Gate passed

Issues
5 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

freedomtan · 2024-11-05T06:20:38Z

@mohitmundhragithub to share the code used to generate this embedding.

…#924) * Add GetConfigValue() * Add custom setting data-format for Core ML * Use GetConfigValue() to get stable_diffusion_seed and stable_diffusion_num_steps

…sion (#930) * Set android:extractNativeLibs="true" * Set android.bundle.enableUncompressedNativeLibs=false

mohitmundhragithub · 2024-11-18T10:51:01Z

@RSMNYS, The function where the time step embedding is generated is shared here: https://github.com/mlcommons/submissions_mobile_v4.1/issues/6#issuecomment-2399205941

From QPM Tutorial package, if we go to this file: <qpm_package>\StableDiffusion1.5\model\example1\redefined_modules\diffusers\models\embeddings.py
get_timestep_embedding( ) function generates the time embeddings.

<qpm_package>\StableDiffusion1.5\model\example3\host_linux_target_android_without_native\qnn_model_execution.ipynb
Above function is called from the top level notebook: get_time_embedding( )

RSMNYS · 2024-11-18T16:58:34Z

@RSMNYS, The function where the time step embedding is generated is shared here: https://github.com/mlcommons/submissions_mobile_v4.1/issues/6#issuecomment-2399205941

From QPM Tutorial package, if we go to this file: <qpm_package>\StableDiffusion1.5\model\example1\redefined_modules\diffusers\models\embeddings.py
get_timestep_embedding( ) function generates the time embeddings.

<qpm_package>\StableDiffusion1.5\model\example3\host_linux_target_android_without_native\qnn_model_execution.ipynb
Above function is called from the top level notebook: get_time_embedding( )

@mohitmundhragithub Shown code is only for the embedding generation. And no operations from the cut part are shown. (fully connected layer, Logistic/Sigmoid activation, and mul operation.

freedomtan · 2024-11-19T06:49:50Z

@anhappdev to share the QPM Stable Diffusion Jupyter notebook with @RSMNYS

Increase minSdkVersion to 30

freedomtan · 2024-11-26T06:31:03Z

@AhmedTElthakeb to provide a small tflite with only 3 operations.

@mohitmundhragithub will try to compare tflite and onnx files.

AhmedTElthakeb · 2024-11-29T22:52:48Z

@AhmedTElthakeb to provide a small tflite with only 3 operations.
sdv15_time_emb_head_int8.zip

RSMNYS · 2024-12-02T20:54:35Z

Hi guys! So I've used the model prepared by Ahmed to generate the embedding file. Now all works good. Here the link to the colab: https://colab.research.google.com/drive/1LG_rC5dlx2CbW2ZF4EamOnCdwtwLQESv?usp=sharing to prepare the embedding file, and the link to the models and embedding file: https://drive.google.com/drive/folders/1CT6VUWwGaTw34Za6dTJE7ptms20NUKH3?usp=sharing

* master: chore: increase Android minSdkVersion from 21 to 30 (#859) fix: resolve crash due to permission denied on Android Play Store version (#930) refactor: use custom setting in Core ML backend to detect NCHW input. (#924) # Conflicts: # mobile_back_tflite/cpp/backend_tflite/stable_diffusion_pipeline.cc

freedomtan · 2024-12-03T06:23:57Z

please use pickle for saving the embedding.

@mohitmundhragithub please help figure out the difference between the one generated by @RSMNYS and Q's.

freedomtan · 2024-12-03T06:27:20Z

once the pickle is done, please make the embedding part of the files to be downloaded for Stable Diffusion for the TFLite backend.

RSMNYS · 2024-12-03T08:21:57Z

please use .pkl for the save the embedding.

@mohitmundhragithub please help figure out the difference between the one generated by @RSMNYS and Q's.

here you can find 2 files in json format to be able to compare numbers at least: https://drive.google.com/drive/folders/1SO1akyvWd2uYz9Xf_u5OGBJLetuW6x5A?usp=sharing

RSMNYS · 2024-12-03T09:54:50Z

guys, I've updated the colab to generate the pkl file. Also adjusted embedding_utils code to parse this. Here you can find the pkl file: https://drive.google.com/file/d/1pDd5wZje1KbIS4JcWzhDx00aN8GmTBpc/view?usp=share_link

RSMNYS · 2024-12-03T14:41:34Z

once the pickle is done, please make the embedding part of the files to be downloaded for Stable Diffusion for the TFLite backend.

@anhappdev I think for this we only need to upload the new UNET model (without embedding operations), and our embedding file. Right? If yes, can you please help upload those 2 files to the storage, from where we download models and other assets:

https://drive.google.com/file/d/1pDd5wZje1KbIS4JcWzhDx00aN8GmTBpc/view?usp=share_link
https://drive.google.com/file/d/1Sf2lcRDjSfg9jgABWWbV5EeWEmXl5CsJ/view?usp=share_link

anhappdev · 2024-12-04T15:45:03Z

@anhappdev I think for this we only need to upload the new UNET model (without embedding operations), and our embedding file. Right? If yes, can you please help upload those 2 files to the storage, from where we download models and other assets:

https://drive.google.com/file/d/1pDd5wZje1KbIS4JcWzhDx00aN8GmTBpc/view?usp=share_link
https://drive.google.com/file/d/1Sf2lcRDjSfg9jgABWWbV5EeWEmXl5CsJ/view?usp=share_link

@RSMNYS Here is the URL for the 2 files you shared
(Please remember to update the checksum in the backend settings)
https://mobile.mlcommons-storage.org/app-resources/models/v4_1/tflite/timestep_embeddings_data.pkl
https://mobile.mlcommons-storage.org/app-resources/models/v4_1/tflite/sd_diffusion_model_dynamic.tflite

And other model files for reference:
https://mobile.mlcommons-storage.org/app-resources/models/v4_1/tflite/sd_decoder_dynamic.tflite
https://mobile.mlcommons-storage.org/app-resources/models/v4_1/tflite/sd_text_encoder_dynamic.tflite

RSMNYS · 2025-01-03T15:48:10Z

@mohitmundhragithub please share the "final" embedding binary file (so that we can double-check it)

unet_time_step_embeddings_20.pkl.txt timestep_steps_20_int32_embedding_1x1280_float32.bin.ts.txt

Sharing the .pkl files and the .bin files generated as described in the steps here: #928 (comment)

Please note that i was unable to upload .pkl and .bin.ts files (github's restriction), so had to append .txt in the filenames. Please remove the .txt extension to use those.

that works good!

* submission-v4.1: feat: add icon and description for Stable Diffusion benchmark (#917) enable stable diffusion in Pixel backend (#936) Update tflite_settings_mtk_mt6989.pbtxt Update QTI backend for submission v4.1 (#13) Applying linter changes Ran make format Update seed and num_steps for TFLite SD task (#16) Addressing review comments Final Submission for code for Qualcomm Add a caption_id to coco_gen dataset (#918) Enable stable_diffusion tests # Conflicts: # flutter/cpp/datasets/coco_gen.cc # mobile_back_apple/dev-utils/Makefile # mobile_back_tflite/cpp/backend_tflite/backend_settings/tflite_settings_android.pbtxt # mobile_back_tflite/cpp/backend_tflite/stable_diffusion_pipeline.h

RSMNYS · 2025-01-06T09:04:55Z

Hi guys! I can run stable diffusion on my android device (Samsung Galaxy S22), but at the end I have 0 as a result, however I see all the steps in logs. Checking why. Also, during the stable diffusion process we are not updating the progress, so it's hard to understand what is going on.

mohitmundhragithub · 2025-01-06T11:37:32Z

Hi guys! I can run stable diffusion on my android device (Samsung Galaxy S22), but at the end I have 0 as a result, however I see all the steps in logs. Checking why. Also, during the stable diffusion process we are not updating the progress, so it's hard to understand what is going on.

can you share the loadgen and logcat logs?

freedomtan · 2025-01-07T06:37:15Z

Let's

make sure this app works well (check the app log directory, @RSMNYS )
verify tflite files are the same with what @AhmedTElthakeb provided (https://github.com/mlcommons/mobile_model_closed/releases/tag/alpha-tflite-v0.3)
after we merge this, add tflite files, .ts, files to both the mobile_models file and the CloudFlare bucket (@anhappdev)
add .pickle and tflite files to the mobile_open repo (https://github.com/mlcommons/mobile_open/releases). @anhappdev
5.0 submission branch

RSMNYS · 2025-01-07T08:33:27Z

Benchmark result.json
here is my benchmark result file. But this for performance only. I think we need the accuracy as well.

sonarqubecloud · 2025-01-07T13:43:37Z

Quality Gate passed

Issues
6 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

anhappdev · 2025-01-13T03:08:37Z

I tested this PR. Here is the output images of the TFLite backend:
https://drive.google.com/drive/folders/1-2OBHMlfF3fbuXcejeIQzPE-2mD5KPsU?usp=sharing

Many images look "wrong" with the color like this one, please check them out.

Noted that the output images are only saved to disk when we run the accuracy test.
The images should be in /storage/emulated/0/Android/data/org.mlcommons.android.mlperfbench/files/logs/<timestamp>/stable_diffusion-accuracy/cocogen_outputs/

mohitmundhragithub · 2025-01-13T06:25:55Z

I tested this PR. Here is the output images of the TFLite backend: https://drive.google.com/drive/folders/1-2OBHMlfF3fbuXcejeIQzPE-2mD5KPsU?usp=sharing

Many images look "wrong" with the color like this one, please check them out.

Noted that the output images are only saved to disk when we run the accuracy test. The images should be in /storage/emulated/0/Android/data/org.mlcommons.android.mlperfbench/files/logs/<timestamp>/stable_diffusion-accuracy/cocogen_outputs/

almost all the images have such artifacts.
seems more like the output is clipped to some max / min values for some pixels.

freedomtan · 2025-01-14T03:05:09Z

I tested this PR. Here is the output images of the TFLite backend: https://drive.google.com/drive/folders/1-2OBHMlfF3fbuXcejeIQzPE-2mD5KPsU?usp=sharing

Many images look "wrong" with the color like this one, please check them out.

Noted that the output images are only saved to disk when we run the accuracy test. The images should be in /storage/emulated/0/Android/data/org.mlcommons.android.mlperfbench/files/logs/<timestamp>/stable_diffusion-accuracy/cocogen_outputs/

Most likely, it's caused by quantization. At least, we know timesteps embedding works as expected.
To further confirm which model (text encoder, unet/diffusion, or image decoder) is problematic, we can check fp32/fp16 text encoder and image decoder first.

freedomtan · 2025-01-14T06:14:20Z

Let's merge this and create a 5.0 submission branch, then work on using fp16 models.

@AhmedTElthakeb please check if you can export fp16 tflite models (https://ai.google.dev/edge/litert/models/post_training_float16_quant).

freedomtan · 2025-01-14T06:32:45Z

By 5.1, qualcomm backend will use the timestep binary file (so that we have a shared timestep source file). @mohitmundhragithub

AhmedTElthakeb · 2025-01-22T23:17:20Z

Here is the fp16 versions release: alpha-tflite-v0.4

anhappdev and others added 4 commits October 15, 2024 13:10

feat: pass task-specific config to backend (#922)

c78b889

* Add TaskConfig.CustomConfig and pass them to backend * Add CustomConfig for main.cc * Use seed and num_steps from CustomConfig for TFLite backend * Replace std::cout with LOG(INFO) * Format files

feat: add ConvertOutputs() API (#927)

fad36c1

* Add ConvertOutputs() API * Add ConvertOutputs() for mobile_back_tflite * Set minimum macos version * Set minimum macos version to 13.1 * Update _kIphoneOnGitHubAction

feat: timestamp-embedding-parser (WIP)

48cb33c

disabled bitcode to be able compile with new XCode

6d9bc91

chore: formatting

fadee08

anhappdev added 2 commits November 12, 2024 12:05

refactor: use custom setting in Core ML backend to detect NCHW input. (…

1bddf37

…#924) * Add GetConfigValue() * Add custom setting data-format for Core ML * Use GetConfigValue() to get stable_diffusion_seed and stable_diffusion_num_steps

fix: resolve crash due to permission denied on Android Play Store ver…

463d974

…sion (#930) * Set android:extractNativeLibs="true" * Set android.bundle.enableUncompressedNativeLibs=false

chore: increase Android minSdkVersion from 21 to 30 (#859)

e0f6813

Increase minSdkVersion to 30

feat: finalized SD pipeline to use embedding from the binary file.

1008c49

refactor: updated embedding_utils to parse pkl file

8527f36

RSMNYS added 3 commits December 3, 2024 11:57

chore: linting

1cec8b3

fix: fixed lint issue in neuron

47c54fa

chore: BUILD cleanup

1c97942

chore: cleanup

4b67590

RSMNYS added 5 commits January 3, 2025 18:03

chore: added links to the sd models and timestep embeddings file

336bd76

chore: add the proper name for the embedding_timesteps file

f222a14

chore: added missed declaration for backend_convert_outputs

b14424e

chore: clang formatting

3e5a5e2

RSMNYS added 3 commits January 7, 2025 10:42

chore: added missed files

809b11d

chore: fixed build file for the pixel backend

ddedee4

chore: bazel formatting

3d58e5a

anhappdev mentioned this pull request Jan 7, 2025

Finalize submission v4.1 #943

Closed

6 tasks

RSMNYS added 2 commits January 7, 2025 14:40

fix: added missed interface implementation for pixel

6170627

chore: clang formatting

f29dccd

anhappdev changed the title ~~Features/timestamp-embedding-parser~~ use time step embedding from file Jan 13, 2025

freedomtan marked this pull request as ready for review January 14, 2025 06:15

freedomtan requested a review from a team as a code owner January 14, 2025 06:15

freedomtan approved these changes Jan 14, 2025

View reviewed changes

RSMNYS merged commit 48654bd into submission-v4.1 Jan 14, 2025
22 checks passed

RSMNYS deleted the features/timestamp-embedding-parser branch January 14, 2025 07:02

github-actions bot locked and limited conversation to collaborators Jan 14, 2025

Conversation

RSMNYS commented Oct 28, 2024

Uh oh!

github-actions bot commented Oct 28, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sonarqubecloud bot commented Oct 28, 2024

Quality Gate passed

Uh oh!

freedomtan commented Nov 5, 2024

Uh oh!

mohitmundhragithub commented Nov 18, 2024

Uh oh!

RSMNYS commented Nov 18, 2024

Uh oh!

freedomtan commented Nov 19, 2024

Uh oh!

freedomtan commented Nov 26, 2024

Uh oh!

AhmedTElthakeb commented Nov 29, 2024

Uh oh!

RSMNYS commented Dec 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

freedomtan commented Dec 3, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

freedomtan commented Dec 3, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

RSMNYS commented Dec 3, 2024

Uh oh!

RSMNYS commented Dec 3, 2024

Uh oh!

RSMNYS commented Dec 3, 2024

Uh oh!

anhappdev commented Dec 4, 2024

Uh oh!

RSMNYS commented Jan 3, 2025

Uh oh!

RSMNYS commented Jan 6, 2025

Uh oh!

mohitmundhragithub commented Jan 6, 2025

Uh oh!

freedomtan commented Jan 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

RSMNYS commented Jan 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sonarqubecloud bot commented Jan 7, 2025

Quality Gate passed

Uh oh!

anhappdev commented Jan 13, 2025

Uh oh!

mohitmundhragithub commented Jan 13, 2025

Uh oh!

freedomtan commented Jan 14, 2025

Uh oh!

freedomtan commented Jan 14, 2025

Uh oh!

freedomtan commented Jan 14, 2025

Uh oh!

Uh oh!

AhmedTElthakeb commented Jan 22, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

github-actions bot commented Oct 28, 2024 •

edited

Loading

RSMNYS commented Dec 2, 2024 •

edited

Loading

freedomtan commented Dec 3, 2024 •

edited

Loading

freedomtan commented Dec 3, 2024 •

edited

Loading

freedomtan commented Jan 7, 2025 •

edited

Loading

RSMNYS commented Jan 7, 2025 •

edited

Loading