Skip to content

Commit cfb62ef

Browse files
🐛 fix lm_head weight mapping (#76)
#### Motivation Fixing an internal issue where `ml6team/keyphrase-generation-t5-small-inspec` and the `t5-small` models were failing due to `RuntimeError: weight lm_head.weight does not exist` error. #### Modifications We added the `lm_head.weight` alias to the list of known aliases for `t5` type models. (It was discovered within the `metadata` of the tensor file during inspection) ``` "lm_head.weight":["decoder.embed_tokens.weight"] ``` #### Result Now we're able to launch `t5` based `ml6team/keyphrase-generation-t5-small-inspec` and `t5-small` models using `tgis_native`! ``` DEPLOYMENT_FRAMEWORK=tgis_native MODEL_NAME=ml6team/keyphrase-generation-t5-small-inspec text-generation-launcher ``` --------- Signed-off-by: Prashant Gupta <[email protected]> Signed-off-by: Joe Runde <[email protected]> Co-authored-by: Joe Runde <[email protected]>
1 parent ea85d72 commit cfb62ef

File tree

1 file changed

+4
-1
lines changed

1 file changed

+4
-1
lines changed

server/text_generation_server/inference_engine/tgis_native.py

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -64,7 +64,10 @@ def __init__(
6464
model_class = BloomForCausalLM
6565

6666
elif model_type == "t5":
67-
aliases = {"shared.weight": ["encoder.embed_tokens.weight", "decoder.embed_tokens.weight"]}
67+
aliases = {
68+
"shared.weight": ["encoder.embed_tokens.weight", "decoder.embed_tokens.weight"],
69+
"lm_head.weight": ["decoder.embed_tokens.weight"],
70+
}
6871
from text_generation_server.models.custom_modeling.t5_modeling import T5ForConditionalGeneration
6972
model_class = T5ForConditionalGeneration
7073

0 commit comments

Comments
 (0)