[Bug] Qwen3-VL-30B-A3B (MoE) fails to start with speculative decoding: missing get_embed_and_head() / set_eagle3_layers_to_capture() in Qwen3VLMoeForConditionalGeneration (sglang v0.5.7)

[JasonNing96]

Checklist

• I searched related issues but found no solution.
• The bug persists in the latest version.
• Issues without environment info and a minimal reproducible demo are hard to resolve and may receive no feedback.
• If this is not a bug report but a general question, please start a discussion at https://github.com/sgl-project/sglang/discussions. Otherwise, it will be closed.
• Please use English. Otherwise, it will be closed.

Describe the bug

Environment

• SGLang: 0.5.7 (docker image / pip package)

• Model (target): Qwen3-VL-30B-A3B-Instruct-AWQ (MoE)

• Draft model: SpecForge_qwen3-vl-model_eagle3

• Two platforms reproduced:

  1. Jetson AGX Orin 64GB (ARM64)
  2. x86_64 server

• dtype: bfloat16

• Context length: 4096

• Note: export SGLANG_DISABLE_CUDNN_CHECK=1 on Jetson

---

Problem Summary

When enabling speculative decoding with Qwen3-VL-30B-A3B (MoE), the server fails to start due to missing methods on the target model wrapper class Qwen3VLMoeForConditionalGeneration:

1. With --speculative-algorithm EAGLE, SGLang crashes in EAGLEWorker because the target model has no get_embed_and_head().

2. With --speculative-algorithm EAGLE3 (Spec V2 path), SGLang crashes during cuda graph init because the target model has no set_eagle3_layers_to_capture().

This happens consistently on both Jetson and x86.

---

Reproduction

A) Jetson AGX 64G (docker, sglang 0.5.7)

Command:

export SGLANG_DISABLE_CUDNN_CHECK=1
python3 -m sglang.launch_server \
  --model-path /home/stardust/zc/models/Qwen3-VL-30B-A3B-Instruct-AWQ \
  --speculative-draft-model-path /home/stardust/zc/models/SpecForge_qwen3-vl-model_eagle3 \
  --tp 1 --port 8007 \
  --enable-torch-compile \
  --context-length 4096 --chunked-prefill-size 4096 \
  --mem-fraction-static 0.6 \
  --trust-remote-code \
  --served-model-name Qwen3_VL_8B \
  --cuda-graph-max-bs 4 \
  --speculative-algorithm EAGLE \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4

Error (key part):

AttributeError: 'Qwen3VLMoeForConditionalGeneration' object has no attribute 'get_embed_and_head'
  File .../sglang/srt/speculative/eagle_worker.py", line 154, in __init__
    embed, head = self.target_worker.model_runner.model.get_embed_and_head()

Also noticed warning:

Unrecognized keys in `rope_scaling` for 'rope_type'='default': {'mrope_interleaved', 'mrope_section'}

---

B) x86_64 (sglang 0.5.7), EAGLE

Command:

python3 -m sglang.launch_server \
  --model-path /home/stardust/zc/models/Qwen3-VL-30B-A3B-Instruct-AWQ/ \
  --speculative-draft-model-path /home/stardust/zc/models/SpecForge_qwen3-vl-model_eagle3/ \
  --tp 1 --port 8007 \
  --context-length 4096 --chunked-prefill-size 4096 \
  --mem-fraction-static 0.6 \
  --trust-remote-code \
  --served-model-name Qwen3_VL_8B \
  --cuda-graph-max-bs 4 \
  --speculative-algorithm EAGLE \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4

Error (same root cause):

AttributeError: 'Qwen3VLMoeForConditionalGeneration' object has no attribute 'get_embed_and_head'

---

C) x86_64 (sglang 0.5.7), EAGLE3

Command:

python3 -m sglang.launch_server \
  --model-path /home/stardust/zc/models/Qwen3-VL-30B-A3B-Instruct-AWQ/ \
  --speculative-draft-model-path /home/stardust/zc/models/SpecForge_qwen3-vl-model_eagle3/ \
  --tp 1 --port 8007 \
  --context-length 4096 --chunked-prefill-size 4096 \
  --mem-fraction-static 0.6 \
  --trust-remote-code \
  --served-model-name Qwen3_VL_8B \
  --cuda-graph-max-bs 4 \
  --speculative-algorithm EAGLE3 \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4

Error (key part):

AttributeError: 'Qwen3VLMoeForConditionalGeneration' object has no attribute 'set_eagle3_layers_to_capture'
  File .../sglang/srt/model_executor/cuda_graph_runner.py", line 353, in __init__
    self.model_runner.model.set_eagle3_layers_to_capture()

---

Expected Behavior

Server should start successfully with speculative decoding enabled (EAGLE / EAGLE3), or at least provide a clear message that Qwen3-VL MoE is not supported by EAGLE/EAGLE3 yet.

---

Additional Context / Suspected Cause

• In the speculative path, workers appear to require target model wrapper to implement:

  • get_embed_and_head() (used by EAGLE worker)
  • set_eagle3_layers_to_capture() (used by EAGLE3 + cuda graph init)

• Qwen3VLMoeForConditionalGeneration currently lacks these methods in v0.5.7.

• There is a merged PR for Qwen2.5-VL + Eagle3 support (PR Qwen2.5-VL eagle3 infer #8801). It seems Qwen3-VL (especially MoE wrapper) may need similar adaptation.

---

Questions

1. Is Qwen3-VL MoE intended to be supported by EAGLE/EAGLE3 in current releases? If not, can we get an explicit compatibility note/error message?
2. If yes, is there an existing branch/PR that adds get_embed_and_head() and set_eagle3_layers_to_capture() (and any related aux hidden state handling) for Qwen3-VL MoE?
3. Any recommended workaround for now (e.g., disable cuda graph / torch compile / use STANDALONE speculative / NGRAM)?

Reproduction

export SGLANG_DISABLE_CUDNN_CHECK=1
python3 -m sglang.launch_server \
  --model-path /home/stardust/zc/models/Qwen3-VL-30B-A3B-Instruct-AWQ \
  --speculative-draft-model-path /home/stardust/zc/models/SpecForge_qwen3-vl-model_eagle3 \
  --tp 1 --port 8007 \
  --enable-torch-compile \
  --context-length 4096 --chunked-prefill-size 4096 \
  --mem-fraction-static 0.6 \
  --trust-remote-code \
  --served-model-name Qwen3_VL_8B \
  --cuda-graph-max-bs 4 \
  --speculative-algorithm EAGLE \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4

Environment

Environment

• SGLang: 0.5.7 (docker image / pip package)

• Model (target): Qwen3-VL-30B-A3B-Instruct-AWQ (MoE)

• Draft model: SpecForge_qwen3-vl-model_eagle3

• Two platforms reproduced:

  1. Jetson AGX Orin 64GB (ARM64)
  2. x86_64 server

• dtype: bfloat16

• Context length: 4096

• Note: export SGLANG_DISABLE_CUDNN_CHECK=1 on Jetson

[JasonNing96]

add：
The model "SpecForge_qwen3-vl-model_eagle3" is the eagle3 trainned by SpecForge from qwen3-vl-30B-A3b MODEL.

[JustinTong0323]

we have #17276, may you check it? Thanks!

[JasonNing96]

Summary

Very thanks for your advice #17276 , the VL function solved our problem.

After aligning to the upstream Qwen3-VL EAGLE3 support changes (the fix that adds required hooks such as get_embed_and_head() / set_eagle3_layers_to_capture() and auxiliary hidden-state handling), the server starts successfully and inference works. However, the EAGLE3 accept rate remains low (~0.25) even on text-only requests. This level of acceptance may significantly reduce the net throughput gain, depending on draft cost and verification overhead.

---

Reproduction (high level)

We run SGLang with:

• target: Qwen3-VL-30B-A3B-Instruct-AWQ
• draft: SpecForge_qwen3-vl-model_eagle3
• speculative: --speculative-algorithm EAGLE3

Representative speculative parameters:

• --speculative-num-steps 3
• --speculative-eagle-topk 1
• --speculative-num-draft-tokens 4

Note: This report focuses on accept rate behavior (not the prior startup crash). The startup crash due to missing methods has been addressed by moving to a codebase that includes the Qwen3-VL EAGLE3 support fix.

---

Observed Behavior

• Server launches and inference works.
• For text-only requests, observed accept rate ≈ 0.25.

---

Expected Behavior

For text-only workloads, we expect a higher accept rate (or guidance on expected accept ranges for this model/draft pair), so that EAGLE3 yields meaningful throughput improvements.

---

Questions for Maintainers / Community

1. Is low accept rate (~0.25) for Qwen3-VL-30B-A3B-Instruct-AWQ (MoE + AWQ) with SpecForge_qwen3-vl-model_eagle3 a known behavior on text-only workloads?
2. Are there recommended parameter ranges for improving acceptance and net speedup (e.g., num_steps, num_draft_tokens, eagle_topk, and accept thresholds)?
3. Could AWQ quantization on the target model significantly reduce acceptance when the draft is bf16/fp16? Any recommended alignment strategy (e.g., quantization consistency, backend settings)?
4. Is there a recommended benchmarking script/metrics to evaluate net gain (e.g., accept length per verify, verify count, draft vs verify time breakdown)?

---

Planned Next Steps (what we will do on our side)

We will further evaluate (sweep) the speculative decoding configuration on our in-house production workloads to improve the accept rate and overall efficiency. If we obtain new findings (e.g., parameter ranges that consistently improve accept rate, workload-specific behaviors, or additional reproducible cases), we will share updates with the community in follow-up comments or a new issue.

---

Optional: Additional Info We Can Provide Upon Request

• Sampling configuration: temperature / top_p / top_k / repetition penalty (and whether draft/target are fully aligned)
• Acceptance statistics beyond rate: average accepted tokens per verify (accept length), number of verify steps, and draft vs target latency breakdown
• A/B comparison on non-AWQ (bf16) target if relevant for isolating quantization effects

[ardenma]

Hi is the acceptance rate low for just text requests or did you try multimodal inputs as well?

Also can you share more about how you trained the speculation model in specforge? I have a PR there as well for training: sgl-project/SpecForge#431 and one key change is that you must use aux layers > 3 to avoid a train/test mismatch related to where/when the deepstack embedding are captured in specforge vs sglang.

The other thing you may want to try is just rebasing the inference PR on top of a release commit to avoid noise in between releases.