Text encoder is loaded all the time and it's taking few minutes

Hi, currently using a workflow where Im using the text encoder umt5-xxl-enc-bf16.safetensors, I noticed that even with high end gpu and actives workers, for every request it's spend 4 minutes to be loaded which slow down considerably the gen time of my workflow. Any solutions ? Thanks a lot !