Skip to content

Commit b7dce49

Browse files
committed
Document Eagle3 auxiliary layer default selection in Llama
Add documentation explaining that get_eagle3_aux_hidden_state_layers() provides default layer selection and that the GPU model runner can override this with values from speculative config for dynamic configuration.
1 parent 69862ba commit b7dce49

File tree

1 file changed

+5
-0
lines changed

1 file changed

+5
-0
lines changed

vllm/model_executor/models/llama.py

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -578,6 +578,11 @@ def set_aux_hidden_state_layers(self, layers: tuple[int, ...]) -> None:
578578
self.model.aux_hidden_state_layers = layers
579579

580580
def get_eagle3_aux_hidden_state_layers(self) -> tuple[int, ...]:
581+
"""Override to return default layers for Llama
582+
583+
Note: The GPU model runner will override this with layers from
584+
the speculative config if available, providing dynamic configuration.
585+
"""
581586
num_layers = len(self.model.layers)
582587
return (2, num_layers // 2, num_layers - 3)
583588

0 commit comments

Comments
 (0)