Poor concept model transfer on target image

[chinembedded]

Hi, Thanks for sharing you work. Very impressive.

To see how the faceswaps work, I tried training my own concept model with 1000 images of myself and face cropped using the following command.

accelerate launch train_dreambooth.py \
  --pretrained_model_name_or_path="CompVis/stable-diffusion-v1-4" \
  --instance_data_dir="my_data" \
  --output_dir="my_model" \
  --instance_prompt="a photo of sks person" \
  --resolution=512 \
  --train_batch_size=1 \
  --gradient_accumulation_steps=1 \
  --gradient_checkpointing \
  --learning_rate=1e-6 \
  --lr_scheduler="constant" \
  --use_8bit_adam \
  --enable_xformers_memory_efficient_attention \
  --max_train_steps=3200 \
  --lr_warmup_steps=0

When I tested the output of the concept model using the following code snippet, I can see my face generated, however, when I use the same model in photoswap as a concept, I cant see a faceswap generate. I see the same target image with some preprocessing applied but no faceswap with my image.

from diffusers import StableDiffusionPipeline
pipe = StableDiffusionPipeline.from_pretrained("./my_model")
pipe.to("cuda")

prompt = "A photo of sks person"
image = pipe(prompt).images[0]

If possible, could you please share your training scripts to understand if I am missing something.