Add Qwen 3.5 model to hub

[laxmareddyp]

Description of the change

This PR address the feature request #2612 issue.

Qwen3.5 Model:

• Represents a departure from traditional Transformer-only designs, moving toward a hybrid "native multimodal" architecture.
• introduces a significant architectural shift from the previous generation by moving toward a native multimodal agent foundation.
• The core innovation is a hybrid design that combines Gated Delta Networks (GDN) with sparse Mixture-of-Experts (MoE), specifically optimized for high-throughput inference and massive context windows up to 1 million tokens.
• It is designed to function as an "all-in-one AI worker" capable of seeing, reading, coding, and planning within a single unified backbone.

Core Architectural Features:

• Hybrid Attention Core:

  • To overcome the quadratic scaling limitations of standard attention, Qwen 3.5 uses a 3:1 hybrid layout. It alternates three Gated DeltaNet layers (linear attention) with one standard Gated Attention layer (full softmax attention).
  • Efficiency: GDN maintains a fixed-size state matrix, keeping compute and memory costs linear relative to sequence length.
  • Structure: The model typically alternates between GDN layers and standard full attention layers in a 3:1 ratio (e.g., three GDN blocks followed by one full attention block).

• Sparse Mixture-of-Experts (MoE): The flagship models utilize high-sparsity routing to maintain massive knowledge capacity with low compute costs.

• Native Multimodality (Early Fusion): Unlike previous versions that used "late fusion" (attaching a separate vision encoder), Qwen3.5 uses early fusion. Text and visual tokens enter the same backbone from the start, allowing the model to process mixed content—like screenshots and code—simultaneously.

Reference

Colab Notebook

Checklist

• I have added all the necessary unit tests for my change.
• I have verified that my change does not break existing code and works with all backends (TensorFlow, JAX, and PyTorch).
• My PR is based on the latest changes of the main branch (if unsure, rebase the code).
• I have followed the Keras Hub Model contribution guidelines in making these changes.
• I have followed the Keras Hub API design guidelines in making these changes.
• I have signed the Contributor License Agreement.

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces the Qwen 3.5 model to Keras Hub, enabling users to leverage its advanced hybrid architecture for causal language modeling. The integration supports its unique combination of full and linear attention mechanisms, designed for high-throughput inference and multimodal capabilities. This addition expands the range of state-of-the-art models available within the Keras ecosystem.

Highlights

• New Model Integration: Integrated the Qwen 3.5 model, a hybrid 'native multimodal' architecture, into Keras Hub, addressing feature request Add Qwen 3.5 to hub #2612.
• Hybrid Architecture Support: Implemented support for Qwen 3.5's core architectural features, including a hybrid attention core (alternating Gated Delta Networks and standard Gated Attention) and sparse Mixture-of-Experts (MoE).
• Gated Delta Net (GDN) Implementation: Added a new Qwen3_5GatedDeltaNet layer for recurrent linear attention, featuring causal Conv1d and delta rule recurrence, which is a key component of Qwen 3.5's efficiency.
• Qwen 3.5 Specific Layers and Components: Introduced dedicated Keras layers for Qwen 3.5 attention, RMS normalization, and a specialized tokenizer, along with a full CausalLM model and preprocessor.
• Hugging Face Checkpoint Conversion: Provided a utility for converting Qwen 3.5 checkpoints from Hugging Face Transformers to the Keras Hub format, including validation tests for tokenizer, model logits, and greedy generation.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

• keras_hub/api/models/init.py
  • Imported new Qwen3.5 backbone, causal LM, preprocessor, and tokenizer classes to the API.
• keras_hub/src/models/qwen3_5/init.py
  • Added initialization file to register Qwen3.5 backbone presets.
• keras_hub/src/models/qwen3_5/qwen3_5_attention.py
  • Added the Qwen3_5Attention layer, implementing grouped-query attention with partial rotary embeddings, sigmoid gating, and optional sliding window.
• keras_hub/src/models/qwen3_5/qwen3_5_backbone.py
  • Added the Qwen3_5Backbone class, defining the hybrid Transformer core architecture for Qwen3.5 models.
• keras_hub/src/models/qwen3_5/qwen3_5_backbone_test.py
  • Added unit tests for the Qwen3_5Backbone to verify basic functionality and parameter counts.
• keras_hub/src/models/qwen3_5/qwen3_5_causal_lm.py
  • Added the Qwen3_5CausalLM class, an end-to-end model for causal language modeling with cache support for autoregressive decoding.
• keras_hub/src/models/qwen3_5/qwen3_5_causal_lm_preprocessor.py
  • Added the Qwen3_5CausalLMPreprocessor class for preparing text data for the Qwen3.5 CausalLM.
• keras_hub/src/models/qwen3_5/qwen3_5_causal_lm_preprocessor_test.py
  • Added unit tests for the Qwen3_5CausalLMPreprocessor to ensure correct tokenization and preprocessing.
• keras_hub/src/models/qwen3_5/qwen3_5_causal_lm_test.py
  • Added unit tests for the Qwen3_5CausalLM, covering basic functionality, generation, and early stopping.
• keras_hub/src/models/qwen3_5/qwen3_5_decoder.py
  • Added the Qwen3_5TransformerDecoder layer, which dispatches between full self-attention and linear attention based on layer type.
• keras_hub/src/models/qwen3_5/qwen3_5_gated_delta_net.py
  • Added the Qwen3_5GatedDeltaNet layer, implementing the recurrent linear attention mechanism with causal Conv1d and delta rule recurrence.
• keras_hub/src/models/qwen3_5/qwen3_5_layernorm.py
  • Added the Qwen3_5LayerNorm class, an RMS normalization layer specific to Qwen3.5.
• keras_hub/src/models/qwen3_5/qwen3_5_presets.py
  • Added an empty backbone_presets dictionary for Qwen3.5 model configurations.
• keras_hub/src/models/qwen3_5/qwen3_5_tokenizer.py
  • Added the Qwen3_5Tokenizer class, a byte-pair encoding (BPE) tokenizer for Qwen3.5 models.
• keras_hub/src/utils/transformers/convert_qwen3_5.py
  • Added a utility script to convert Hugging Face Qwen3.5 model configurations and weights to Keras Hub format.
• keras_hub/src/utils/transformers/preset_loader.py
  • Imported the convert_qwen3_5 module.
  • Registered the qwen3_5 model type with its corresponding converter in the PresetLoader.
• tools/checkpoint_conversion/convert_qwen3_5_checkpoints.py
  • Added a script for converting and validating Qwen3.5 checkpoints from Hugging Face, including comparisons for tokenizer, model logits, and generated text.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces the Qwen3.5 model, a hybrid attention model, to KerasHub. The changes include the backbone, causal language model task, preprocessor, tokenizer, and associated conversion scripts and tests. While the overall structure aligns with the repository's standards, I've identified several issues that need to be addressed. There are two critical issues: the presets file is empty, which makes the model non-functional, and there's a bug in the ops.split usage within the GatedDeltaNet implementation that will cause a runtime error. Additionally, there's a significant performance concern with the recurrent implementation of the delta rule, which will be inefficient for training. Finally, there is a consistent violation of the repository's class naming conventions across all new model components.

[gemini-code-assist]

There is a bug in how ops.split is used to separate the fused QKV projection. The num_or_size_splits argument is [self.key_dim, self.key_dim * 2], which will produce two tensors. However, the code attempts to unpack the result into three variables (query, key, value), which will cause a runtime error. The correct split sizes should be [self.key_dim, self.key_dim, self.value_dim] to match the three output tensors.

Suggested change

	query, key, value = ops.split(
	mixed_qkv,
	[self.key_dim, self.key_dim * 2],
	axis=-1,
	)
	query, key, value = ops.split(
	mixed_qkv,
	[self.key_dim, self.key_dim, self.value_dim],
	axis=-1,
	)

[gemini-code-assist]

The backbone_presets dictionary is empty. This makes the model unusable as there are no pre-trained configurations to load. The conversion script tools/checkpoint_conversion/convert_qwen3_5_checkpoints.py should be run to generate the presets, and the resulting qwen3_5_presets.py file should be included in this pull request. Without presets, this model cannot be instantiated via from_preset().

References

1. Preset files are required to define model configurations, descriptions, and weights URLs. An empty preset file makes the model unusable. ^(link)

[gemini-code-assist]

The implementation of _chunk_gated_delta_rule uses a recurrent loop over the sequence length. As noted in the comment, this is not optimized and will be very slow during training, especially with long sequences. For a model intended for production use, this should be implemented in a parallel fashion (e.g., using a parallel scan) to be efficient for training. The current implementation will likely lead to significant performance bottlenecks.

References

1. Code should use efficient algorithms. A sequential loop over the sequence length for an attention-like mechanism is inefficient for training on modern accelerators. ^(link)

[gemini-code-assist]

The class name Qwen3_5Backbone and other related classes in this PR (e.g., Qwen3_5CausalLM, Qwen3_5Attention) do not follow the repository's naming conventions. According to the style guide, class names should be in CapWords format (PascalCase) and follow the pattern <ModelName><ComponentType>. The use of an underscore in Qwen3_5 is inconsistent with other models in the repository like DebertaV3Backbone or Qwen3MoeBackbone. The name should be Qwen35Backbone. Please apply this change to all new Qwen3_5* classes introduced in this pull request.

Suggested change

	class Qwen3_5Backbone(Backbone):
	class Qwen35Backbone(Backbone):

References

1. Class names must use CapWords (PascalCase) and follow the pattern <ModelName><ComponentType>. The underscore in Qwen3_5 violates this convention. ^(link)