model added qwen3_omni

[lukiod]

Description of the change

ADDED QWEN3_OMNI IN THE MDOEL DIR

Reference

#2413

Colab Notebook

Checklist

• I have added all the necessary unit tests for my change.
• I have verified that my change does not break existing code and works with all backends (TensorFlow, JAX, and PyTorch).
• My PR is based on the latest changes of the main branch (if unsure, rebase the code).
• I have followed the Keras Hub Model contribution guidelines in making these changes.
• I have followed the Keras Hub API design guidelines in making these changes.
• I have signed the Contributor License Agreement.

[gemini-code-assist]

Summary of Changes

Hello @lukiod, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request integrates the Qwen3-Omni Mixture of Experts (MoE) model into the Keras Hub. The primary goal is to expand the library's capabilities with a new, advanced multimodal Transformer architecture. The changes involve introducing the foundational components for this model, including specialized attention mechanisms, MoE routing, and custom normalization layers, alongside a predefined 7B model preset and corresponding unit tests to ensure robust functionality.

Highlights

• New Model Addition: The Qwen3-Omni Mixture of Experts (MoE) model has been added to the Keras Hub, introducing a powerful multimodal architecture.
• Core Components Implemented: New custom layers for multi-head attention (Qwen3OmniMoeAttention), sparse MoE blocks (Qwen3OmniMoeSparseMoeBlock), and layer normalization (Qwen3OmniMoeLayerNorm) have been implemented specifically for the Qwen3-Omni MoE model.
• Multimodal Capabilities: The Qwen3-Omni MoE backbone is designed to support text, audio, and vision inputs, with detailed configurations for each modality included in the model presets.
• Preset Configuration: A 'qwen3_omni_moe_7b' preset has been defined, providing a ready-to-use configuration for a 7 billion parameter version of the model, complete with multimodal settings.
• Comprehensive Testing: Unit tests have been added for the Qwen3OmniMoeBackbone to verify its basic functionality, preset loading, and caching mechanisms.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces the Qwen3OmniMoe model. While the contribution is valuable, the current implementation has several critical issues that need to be addressed. These include incorrect cache handling for generation, bugs in the Mixture-of-Experts (MoE) layer implementation, incorrect layer normalization, and a testing file that does not adhere to the repository's standards. I've provided detailed comments and suggestions to fix these issues.