Hires Fix and TI - man to woman under txt2img

[mykeehu]

I have problems with Hires Fix when using txt2img when using TI. The bodies and faces in the image are fine when there are two of them: one female and one male. The woman is formatted with TI.
The Hires Fix above 0.3 reformats the male to female for TI, i.e. duplicates the person. The higher the value of Hires Fix, the more it feminizes the man, above 0.5 it completely transforms him. However, sometimes you need this much noise to "restore" the fingers and other body parts.
This happens regardless of what Hires Fix model I use, Latent or 4x_foolhardy_Remacri, it will reshape the latent the same way. How can I lock the Hires Fix so that it does not reshape the man into a woman?