How to change part of the original image with image prompt?

[gagbaghdas]

Hey guys. I'm trying to do the following:
I have an image of the man , and i want to change his t-shirt with the given hoody image.
The problem in my current result is that it changes the whole image.

I'm using img2img , IP-Adapter for ControlNet Unit with independent control image without any tex prompt:
Maybe there is something I missed or don't know yet? please help:
Here are the man image and the hoody image:

And here is my current result:

P.S. I can achieve the result with Inpaint , but I'm trying to do the same without it.