Commit 85a754a
authored
Support hybrid attention architectures in LayerWrapper (#2367)
## Summary
Allow `LayerWrapper` to handle models with hybrid layer types (e.g.,
Qwen3.5) where some decoder layers use linear attention instead of
standard self-attention.
## Problem
Qwen3.5 is a hybrid VL model with 24 decoder layers — 18 use
GatedDeltaNet linear attention (`linear_attn` sub-module) and 6 use
standard full attention (`self_attn`). When Olive's
`SelectiveMixedPrecision` or `GPTQ` passes wrap each layer with
`LayerWrapper`, the constructor calls:
```python
self.attn, self.attn_name = get_submodules(
layer, self.ATTENTION, self.model_type, return_name=True
)
```
This raises ```ValueError```for GatedDeltaNet layers since they don't
have a ```self_attn``` attribute.
Fix
Pass ```fail_on_not_found=False``` to the attention sub-module lookup in
```LayerWrapper.__init__```:
```
- self.attn, self.attn_name = get_submodules(
- layer, self.ATTENTION, self.model_type, return_name=True
- )
+ # Use fail_on_not_found=False to support hybrid architectures (e.g., Qwen3.5)
+ # where some layers use linear attention instead of standard self-attention
+ self.attn, self.attn_name = get_submodules(
+ layer, self.ATTENTION, self.model_type, return_name=True, fail_on_not_found=False
+ )
```
When a layer doesn't have a standard attention module, ```self.attn```
is set to None and the calibration passes gracefully skip
attention-specific quantization for that layer while still processing
the MLP.1 parent 1bfa214 commit 85a754a
1 file changed
+11
-1
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
134 | 134 | | |
135 | 135 | | |
136 | 136 | | |
137 | | - | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
138 | 142 | | |
139 | 143 | | |
140 | 144 | | |
| |||
144 | 148 | | |
145 | 149 | | |
146 | 150 | | |
| 151 | + | |
| 152 | + | |
147 | 153 | | |
148 | 154 | | |
149 | 155 | | |
| |||
153 | 159 | | |
154 | 160 | | |
155 | 161 | | |
| 162 | + | |
| 163 | + | |
156 | 164 | | |
157 | 165 | | |
158 | 166 | | |
| |||
274 | 282 | | |
275 | 283 | | |
276 | 284 | | |
| 285 | + | |
| 286 | + | |
277 | 287 | | |
278 | 288 | | |
279 | 289 | | |
| |||
0 commit comments