Fix Whisper

[jackzhxng]

Fixes Whisper export + misc seq2seq-related cleanups, which was broken a while back due to transformers changes

Runs on both portable and XNNPack.

Stack from (oldest at bottom):

• Portable recipe exports into one pte with multiple methods #171
• -> Fix Whisper #167
• Fix T5 #172

[jackzhxng]

Note - ignore the CI test failures, they are unrelated. If you click into the logs you can see that tests are passing until it runs out of disk space, additionally I've verified these test locally as well.

[dylanschoenmakers]

With this change ASR models are now exported as a single model.pte with encoder and decoder methods instead of a forward correct? Is there a way to still export them as separate encoder/decoder models so that it works with the current version of react-native-executorch?

[jackzhxng]

@dylanschoenmakers yes that's correct, reason is that we usually recommend the one model / multiple method approach since there's less overhead.

cc @chmjkb is this something you'd be able to update for react-native-excecutorch?

[chmjkb]

@jackzhxng, sure we'll update it as soon as we can, thanks for fixing this! Any ideas on what the perf improvements should be?

[jackzhxng]

cc @larryliu0820

[larryliu0820]

@jackzhxng, sure we'll update it as soon as we can, thanks for fixing this! Any ideas on what the perf improvements should be?

I'm getting 3.4 tok/s on whisper-small (not tiny)

[larryliu0820]

With this change ASR models are now exported as a single model.pte with encoder and decoder methods instead of a forward correct? Is there a way to still export them as separate encoder/decoder models so that it works with the current version of react-native-executorch?

I think having both methods in one file makes sure that they can work together. We also recognize it makes the model artifact much bigger, but I believe using program data separation can mitigate this issue.

[larryliu0820]

@jackzhxng, sure we'll update it as soon as we can, thanks for fixing this! Any ideas on what the perf improvements should be?

I'm getting 3.4 tok/s on whisper-small (not tiny)

Oh if you are curious about perf improvement between separate files and one file, I don’t think there will be significant perf improvement, but just the benefit I commented earlier.