Merge pull request #885 from roboflow/florence2_speedups

isaacrob · web-flow · commit 80af3da6a068 · 2024-12-17T12:13:04.000-07:00
merge lora into base model for 3x speedup
diff --git a/inference/models/transformers/transformers.py b/inference/models/transformers/transformers.py
@@ -252,6 +252,8 @@ def initialize_model(self):
             .to(self.dtype)
         )
 
+        self.model.merge_and_unload()
+
         self.processor = self.processor_class.from_pretrained(
             model_load_id, revision=revision, cache_dir=cache_dir, token=token
         )

Original file line number	Diff line number	Diff line change
`@@ -252,6 +252,8 @@ def initialize_model(self):`
`252`	`252`	`.to(self.dtype)`
`253`	`253`	`)`
`254`	`254`
	`255`	`+ self.model.merge_and_unload()`
	`256`	`+`
`255`	`257`	`self.processor = self.processor_class.from_pretrained(`
`256`	`258`	`model_load_id, revision=revision, cache_dir=cache_dir, token=token`
`257`	`259`	`)`