Add Qwen3-VL vision-language model support

[nyo16]

Summary

This PR adds full support for the Qwen3-VL vision-language model family, enabling image-to-text generation with Bumblebee.

Model: Qwen/Qwen3-VL-2B-Instruct (and other sizes)
Architecture: Qwen3VLForConditionalGeneration

Features

Vision Encoder (Bumblebee.Vision.Qwen3VLVision)

• 3D convolution patch embedding (supports video temporal dimension)
• 2D spatial rotary position embeddings for accurate spatial understanding
• Bilinear interpolation for position embeddings (handles variable image sizes)
• Patch merger with spatial reduction (2x2 → 1)
• DeepStack feature extraction from layers [5, 11, 17]

Text Decoder

• Based on Qwen3 architecture with QK-norm
• Visual token substitution (replaces image placeholder tokens with vision embeddings)
• DeepStack injection at decoder layers [0, 1, 2]
• Full rotary position embedding support

Featurizer (Bumblebee.Vision.Qwen3VLFeaturizer)

• Image preprocessing with configurable resize
• Automatic padding to patch-aligned dimensions
• Support for both images and video frames
• Outputs flattened patches: {num_patches, channels * temporal * patch_h * patch_w}

DeepStack Implementation

DeepStack provides multi-scale visual information by:

1. Extracting hidden states from vision encoder layers [5, 11, 17] (1-indexed)
2. Passing each through separate merger MLPs with postshuffle norm (norm AFTER spatial merge)
3. Injecting features into text decoder at layers [0, 1, 2]
4. Formula: hidden_states[visual_mask] += deepstack_features[layer_idx]

Infrastructure Changes

• Added post_block_hook option to Layers.Transformer.blocks for per-layer injection

Files Changed

New Files

• lib/bumblebee/multimodal/qwen3_vl.ex - Main VL model
• lib/bumblebee/vision/qwen3_vl_vision.ex - Vision encoder
• lib/bumblebee/vision/qwen3_vl_featurizer.ex - Image preprocessing
• test/bumblebee/multimodal/qwen3_vl_test.exs - Tests
• notebooks/qwen3_vl.livemd - Usage examples

Modified Files

• lib/bumblebee.ex - Model/featurizer registrations
• lib/bumblebee/layers/transformer.ex - Added post_block_hook option

Test Results

# Unit test with tiny model
mix test test/bumblebee/multimodal/qwen3_vl_test.exs
1 test, 0 failures

# Real model test with image (448x448)
Image: 448x448x3
Patches: 784 → 196 visual tokens (after merge)
Generated: "This image shows a close-up of a single, small, dark-colored object..."
✓ DeepStack injection test with real image PASSED!

Usage Example

{:ok, model_info} = Bumblebee.load_model({:hf, "Qwen/Qwen3-VL-2B-Instruct"}, type: :bf16)
{:ok, tokenizer} = Bumblebee.load_tokenizer({:hf, "Qwen/Qwen3-VL-2B-Instruct"})
{:ok, featurizer} = Bumblebee.load_featurizer({:hf, "Qwen/Qwen3-VL-2B-Instruct"},
  module: Bumblebee.Vision.Qwen3VLFeaturizer)

# Load and process image
image = StbImage.read_file!("photo.jpg")
image_inputs = Bumblebee.apply_featurizer(featurizer, image)

# Build prompt with image placeholder
prompt = """
<|im_start|>user
<|vision_start|><|image_pad|><|vision_end|>Describe this image.<|im_end|>
<|im_start|>assistant
"""

text_inputs = Bumblebee.apply_tokenizer(tokenizer, prompt)
inputs = Map.merge(text_inputs, image_inputs)

# Run inference
outputs = Axon.predict(model_info.model, model_info.params, inputs)

Parameter Loading

All parameters load correctly with no warnings:

• Vision encoder: patch_embed, pos_embed, blocks.{0-23}, merger
• DeepStack mergers: deepstack_merger_list.{0-2} (9 params total)
• Text decoder: embedder, decoder.blocks.{0-27}, output_norm, lm_head

References

• Qwen3-VL on HuggingFace
• Qwen3-VL Technical Report