Skip to content

Commit 75c79e9

Browse files
committed
Document Eagle3 auxiliary layer default selection in Llama
Add documentation explaining that get_eagle3_aux_hidden_state_layers() provides default layer selection and that the GPU model runner can override this with values from speculative config for dynamic configuration. Signed-off-by: rahul-tuli <[email protected]>
1 parent 4d812bb commit 75c79e9

File tree

1 file changed

+5
-0
lines changed

1 file changed

+5
-0
lines changed

vllm/model_executor/models/llama.py

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -578,6 +578,11 @@ def set_aux_hidden_state_layers(self, layers: tuple[int, ...]) -> None:
578578
self.model.aux_hidden_state_layers = layers
579579

580580
def get_eagle3_aux_hidden_state_layers(self) -> tuple[int, ...]:
581+
"""Override to return default layers for Llama
582+
583+
Note: The GPU model runner will override this with layers from
584+
the speculative config if available, providing dynamic configuration.
585+
"""
581586
num_layers = len(self.model.layers)
582587
return (2, num_layers // 2, num_layers - 3)
583588

0 commit comments

Comments
 (0)