About converting from custom SD to Diffusers

[alpkabac]

What would you recommend for someone that tries to use dreambooth with a fine-tuned model such a NAI Diffusion? The conversion does not seem to work.

[TheLastBen]

what type of error did you get ?

[alpkabac]

I always get this error:

Some weights of the model checkpoint at openai/clip-vit-large-patch14 were not used when initializing CLIPTextModel: ['vision_model.encoder.layers.8.mlp.fc1.bias', 'vision_model.encoder.layers.10.layer_norm1.bias', 'vision_model.encoder.layers.5.self_attn.q_proj.bias', 'vision_model.encoder.layers.15.layer_norm1.weight', 'vision_model.encoder.layers.6.mlp.fc2.bias', 'vision_model.encoder.layers.7.self_attn.q_proj.bias', 'vision_model.encoder.layers.16.mlp.fc1.bias', 'vision_model.encoder.layers.11.self_attn.k_proj.bias', 'vision_model.encoder.layers.1.layer_norm1.weight', 'vision_model.encoder.layers.10.mlp.fc1.weight', 'vision_model.encoder.layers.6.self_attn.v_proj.bias', 'vision_model.encoder.layers.7.mlp.fc2.weight', 'vision_model.encoder.layers.13.mlp.fc1.bias', 'vision_model.encoder.layers.5.mlp.fc2.bias', 'vision_model.encoder.layers.0.self_attn.k_proj.weight', 'vision_model.encoder.layers.5.mlp.fc1.weight', 'vision_model.encoder.layers.2.self_attn.q_proj.weight', 'vision_model.encoder.layers.17.self_attn.q_proj.bias', 'vision_model.encoder.layers.23.mlp.fc2.weight', 'vision_model.encoder.layers.10.self_attn.k_proj.weight', 'vision_model.pre_layrnorm.bias', 'vision_model.encoder.layers.7.self_attn.q_proj.weight', 'vision_model.encoder.layers.2.layer_norm2.weight', 'vision_model.encoder.layers.15.self_attn.k_proj.weight', 'vision_model.encoder.layers.7.layer_norm1.weight', 'vision_model.post_layernorm.bias', 'vision_model.encoder.layers.5.layer_norm2.weight', 'vision_model.encoder.layers.23.mlp.fc1.bias', 'vision_model.encoder.layers.17.mlp.fc2.weight', 'vision_model.encoder.layers.1.mlp.fc2.bias', 'vision_model.encoder.layers.6.layer_norm1.bias', 'vision_model.encoder.layers.7.layer_norm1.bias', 'vision_model.encoder.layers.16.self_attn.k_proj.weight', 'vision_model.encoder.layers.20.layer_norm2.bias', 'vision_model.encoder.layers.11.mlp.fc2.bias', 'vision_model.encoder.layers.15.mlp.fc2.weight', 'vision_model.encoder.layers.5.self_attn.out_proj.bias', 'vision_model.encoder.layers.19.mlp.fc1.bias', 'vision_model.encoder.layers.18.self_attn.k_proj.weight', 'vision_model.encoder.layers.5.layer_norm1.bias', 'vision_model.encoder.layers.9.layer_norm1.weight', 'vision_model.encoder.layers.0.layer_norm2.weight', 'vision_model.encoder.layers.12.self_attn.out_proj.weight', 'vision_model.encoder.layers.16.self_attn.out_proj.weight', 'vision_model.encoder.layers.19.self_attn.k_proj.weight', 'vision_model.encoder.layers.22.layer_norm2.weight', 'vision_model.encoder.layers.7.mlp.fc1.weight', 'vision_model.encoder.layers.1.self_attn.k_proj.bias', 'vision_model.encoder.layers.15.self_attn.q_proj.bias', 'vision_model.encoder.layers.21.self_attn.k_proj.bias', 'vision_model.encoder.layers.21.self_attn.k_proj.weight', 'vision_model.encoder.layers.0.self_attn.q_proj.bias', 'vision_model.encoder.layers.14.self_attn.out_proj.weight', 'vision_model.encoder.layers.2.self_attn.k_proj.weight', 'vision_model.encoder.layers.1.mlp.fc1.weight', 'vision_model.encoder.layers.17.self_attn.v_proj.weight', 'vision_model.encoder.layers.2.layer_norm1.weight', 'vision_model.encoder.layers.1.self_attn.out_proj.bias', 'vision_model.encoder.layers.14.layer_norm2.bias', 'vision_model.encoder.layers.21.self_attn.q_proj.weight', 'vision_model.encoder.layers.3.layer_norm1.bias', 'vision_model.encoder.layers.18.self_attn.out_proj.weight', 'vision_model.encoder.layers.9.self_attn.v_proj.bias', 'vision_model.encoder.layers.3.self_attn.k_proj.weight', 'vision_model.encoder.layers.15.layer_norm2.bias', 'vision_model.encoder.layers.17.mlp.fc1.weight', 'vision_model.encoder.layers.9.layer_norm2.bias', 'vision_model.encoder.layers.3.layer_norm2.bias', 'vision_model.encoder.layers.19.self_attn.v_proj.weight', 'logit_scale', 'vision_model.encoder.layers.10.self_attn.q_proj.bias', 'vision_model.encoder.layers.21.mlp.fc1.weight', 'vision_model.encoder.layers.12.mlp.fc2.bias', 'vision_model.encoder.layers.11.layer_norm2.bias', 'vision_model.encoder.layers.16.self_attn.out_proj.bias', 'vision_model.encoder.layers.8.self_attn.out_proj.bias', 'vision_model.encoder.layers.15.self_attn.out_proj.weight', 'vision_model.encoder.layers.11.self_attn.v_proj.bias', 'vision_model.encoder.layers.20.mlp.fc2.weight', 'vision_model.encoder.layers.23.self_attn.out_proj.weight', 'vision_model.encoder.layers.0.self_attn.v_proj.weight', 'vision_model.encoder.layers.18.mlp.fc2.weight', 'vision_model.encoder.layers.13.mlp.fc2.weight', 'vision_model.encoder.layers.7.self_attn.v_proj.weight', 'vision_model.encoder.layers.7.mlp.fc1.bias', 'vision_model.encoder.layers.20.layer_norm1.bias', 'vision_model.encoder.layers.9.self_attn.out_proj.bias', 'vision_model.encoder.layers.18.layer_norm2.weight', 'vision_model.encoder.layers.8.self_attn.k_proj.bias', 'vision_model.encoder.layers.16.self_attn.q_proj.bias', 'vision_model.encoder.layers.6.self_attn.out_proj.bias', 'vision_model.encoder.layers.0.mlp.fc1.bias', 'vision_model.encoder.layers.19.self_attn.q_proj.bias', 'vision_model.encoder.layers.12.self_attn.v_proj.bias', 'vision_model.encoder.layers.22.self_attn.k_proj.weight', 'vision_model.encoder.layers.3.mlp.fc1.bias', 'vision_model.encoder.layers.19.mlp.fc2.weight', 'vision_model.encoder.layers.23.layer_norm2.weight', 'vision_model.encoder.layers.23.self_attn.k_proj.bias', 'vision_model.embeddings.position_ids', 'vision_model.encoder.layers.4.self_attn.q_proj.bias', 'vision_model.encoder.layers.2.self_attn.out_proj.weight', 'vision_model.encoder.layers.18.self_attn.k_proj.bias', 'vision_model.encoder.layers.9.layer_norm1.bias', 'vision_model.encoder.layers.6.layer_norm2.weight', 'vision_model.encoder.layers.7.mlp.fc2.bias', 'vision_model.encoder.layers.9.mlp.fc2.weight', 'vision_model.encoder.layers.20.mlp.fc2.bias', 'vision_model.encoder.layers.4.self_attn.k_proj.bias', 'vision_model.encoder.layers.20.self_attn.q_proj.weight', 'vision_model.encoder.layers.14.self_attn.q_proj.bias', 'vision_model.encoder.layers.20.self_attn.k_proj.weight', 'vision_model.encoder.layers.3.layer_norm2.weight', 'vision_model.encoder.layers.5.mlp.fc1.bias', 'vision_model.encoder.layers.3.self_attn.v_proj.bias', 'vision_model.encoder.layers.11.self_attn.out_proj.weight', 'vision_model.encoder.layers.11.layer_norm1.weight', 'vision_model.encoder.layers.12.self_attn.out_proj.bias', 'vision_model.encoder.layers.3.mlp.fc1.weight', 'vision_model.encoder.layers.11.layer_norm1.bias', 'vision_model.encoder.layers.14.self_attn.v_proj.bias', 'vision_model.encoder.layers.0.self_attn.out_proj.bias', 'vision_model.encoder.layers.4.self_attn.k_proj.weight', 'vision_model.encoder.layers.2.self_attn.q_proj.bias', 'vision_model.encoder.layers.7.self_attn.out_proj.bias', 'vision_model.encoder.layers.14.self_attn.k_proj.bias', 'vision_model.encoder.layers.19.self_attn.out_proj.bias', 'vision_model.encoder.layers.17.self_attn.v_proj.bias', 'vision_model.encoder.layers.3.self_attn.q_proj.weight', 'vision_model.encoder.layers.0.self_attn.out_proj.weight', 'vision_model.embeddings.class_embedding', 'vision_model.encoder.layers.13.self_attn.k_proj.bias', 'vision_model.encoder.layers.15.mlp.fc2.bias', 'vision_model.encoder.layers.2.mlp.fc1.bias', 'vision_model.encoder.layers.10.mlp.fc2.weight', 'vision_model.encoder.layers.0.self_attn.q_proj.weight', 'vision_model.encoder.layers.23.self_attn.q_proj.bias', 'vision_model.encoder.layers.17.self_attn.k_proj.bias', 'vision_model.encoder.layers.21.mlp.fc1.bias', 'vision_model.encoder.layers.17.mlp.fc2.bias', 'vision_model.encoder.layers.0.self_attn.v_proj.bias', 'vision_model.encoder.layers.4.self_attn.v_proj.bias', 'vision_model.encoder.layers.13.self_attn.k_proj.weight', 'vision_model.encoder.layers.9.mlp.fc2.bias', 'vision_model.encoder.layers.12.mlp.fc1.bias', 'vision_model.encoder.layers.7.self_attn.v_proj.bias', 'vision_model.encoder.layers.19.self_attn.q_proj.weight', 'vision_model.encoder.layers.16.self_attn.k_proj.bias', 'vision_model.encoder.layers.13.self_attn.out_proj.bias', 'vision_model.encoder.layers.6.self_attn.v_proj.weight', 'vision_model.encoder.layers.5.self_attn.v_proj.weight', 'vision_model.encoder.layers.14.self_attn.q_proj.weight', 'vision_model.encoder.layers.21.self_attn.out_proj.weight', 'vision_model.encoder.layers.23.self_attn.v_proj.weight', 'vision_model.encoder.layers.10.layer_norm2.bias', 'vision_model.encoder.layers.6.layer_norm2.bias', 'vision_model.encoder.layers.11.self_attn.k_proj.weight', 'vision_model.encoder.layers.15.self_attn.v_proj.bias', 'vision_model.encoder.layers.10.mlp.fc2.bias', 'vision_model.encoder.layers.22.self_attn.v_proj.bias', 'vision_model.encoder.layers.18.self_attn.q_proj.weight', 'vision_model.encoder.layers.15.self_attn.q_proj.weight', 'vision_model.post_layernorm.weight', 'vision_model.encoder.layers.8.layer_norm1.weight', 'vision_model.encoder.layers.16.layer_norm2.weight', 'vision_model.encoder.layers.20.mlp.fc1.weight', 'vision_model.encoder.layers.13.layer_norm2.weight', 'vision_model.encoder.layers.22.self_attn.q_proj.weight', 'vision_model.encoder.layers.11.self_attn.v_proj.weight', 'vision_model.encoder.layers.5.mlp.fc2.weight', 'vision_model.encoder.layers.8.self_attn.k_proj.weight', 'vision_model.encoder.layers.13.layer_norm1.bias', 'vision_model.encoder.layers.9.self_attn.out_proj.weight', 'vision_model.encoder.layers.22.mlp.fc1.weight', 'vision_model.encoder.layers.9.mlp.fc1.weight', 'vision_model.encoder.layers.6.self_attn.out_proj.weight', 'vision_model.encoder.layers.13.mlp.fc2.bias', 'vision_model.encoder.layers.15.self_attn.k_proj.bias', 'vision_model.encoder.layers.6.self_attn.q_proj.weight', 'vision_model.encoder.layers.18.layer_norm1.weight', 'vision_model.encoder.layers.22.mlp.fc2.bias', 'vision_model.encoder.layers.11.mlp.fc1.weight', 'vision_model.encoder.layers.13.self_attn.v_proj.bias', 'vision_model.encoder.layers.4.mlp.fc1.weight', 'vision_model.encoder.layers.17.self_attn.q_proj.weight', 'vision_model.encoder.layers.11.mlp.fc1.bias', 'vision_model.encoder.layers.0.mlp.fc1.weight', 'vision_model.encoder.layers.8.layer_norm1.bias', 'vision_model.encoder.layers.18.mlp.fc1.weight', 'vision_model.encoder.layers.9.layer_norm2.weight', 'vision_model.encoder.layers.16.mlp.fc1.weight', 'vision_model.encoder.layers.17.self_attn.out_proj.bias', 'vision_model.encoder.layers.6.self_attn.k_proj.weight', 'vision_model.encoder.layers.2.self_attn.out_proj.bias', 'vision_model.encoder.layers.1.self_attn.q_proj.weight', 'vision_model.encoder.layers.1.mlp.fc2.weight', 'vision_model.encoder.layers.20.self_attn.v_proj.weight', 'vision_model.encoder.layers.5.layer_norm1.weight', 'vision_model.encoder.layers.3.mlp.fc2.weight', 'vision_model.encoder.layers.1.self_attn.v_proj.bias', 'vision_model.encoder.layers.20.self_attn.q_proj.bias', 'vision_model.encoder.layers.3.layer_norm1.weight', 'vision_model.encoder.layers.21.layer_norm2.bias', 'vision_model.encoder.layers.14.layer_norm2.weight', 'vision_model.encoder.layers.22.self_attn.q_proj.bias', 'vision_model.encoder.layers.10.self_attn.v_proj.bias', 'vision_model.encoder.layers.22.mlp.fc2.weight', 'vision_model.encoder.layers.4.layer_norm1.bias', 'vision_model.encoder.layers.2.self_attn.k_proj.bias', 'vision_model.encoder.layers.8.self_attn.v_proj.weight', 'vision_model.encoder.layers.11.self_attn.q_proj.bias', 'vision_model.encoder.layers.1.self_attn.k_proj.weight', 'vision_model.encoder.layers.19.mlp.fc1.weight', 'vision_model.encoder.layers.18.self_attn.v_proj.weight', 'vision_model.encoder.layers.22.self_attn.v_proj.weight', 'vision_model.encoder.layers.16.layer_norm2.bias', 'vision_model.encoder.layers.8.self_attn.v_proj.bias', 'vision_model.encoder.layers.9.self_attn.v_proj.weight', 'vision_model.encoder.layers.11.layer_norm2.weight', 'vision_model.encoder.layers.2.layer_norm1.bias', 'vision_model.encoder.layers.5.self_attn.out_proj.weight', 'vision_model.encoder.layers.10.self_attn.out_proj.weight', 'vision_model.encoder.layers.20.mlp.fc1.bias', 'vision_model.encoder.layers.12.self_attn.v_proj.weight', 'vision_model.encoder.layers.17.self_attn.out_proj.weight', 'vision_model.encoder.layers.1.layer_norm2.weight', 'vision_model.encoder.layers.3.self_attn.v_proj.weight', 'vision_model.encoder.layers.0.layer_norm1.weight', 'vision_model.encoder.layers.14.self_attn.k_proj.weight', 'vision_model.encoder.layers.22.self_attn.out_proj.bias', 'vision_model.encoder.layers.0.self_attn.k_proj.bias', 'vision_model.encoder.layers.4.layer_norm2.weight', 'vision_model.encoder.layers.14.layer_norm1.bias', 'vision_model.encoder.layers.12.layer_norm1.weight', 'vision_model.encoder.layers.16.mlp.fc2.bias', 'vision_model.encoder.layers.4.layer_norm1.weight', 'vision_model.encoder.layers.12.layer_norm2.bias', 'vision_model.encoder.layers.23.self_attn.out_proj.bias', 'vision_model.encoder.layers.21.mlp.fc2.bias', 'vision_model.encoder.layers.21.layer_norm2.weight', 'vision_model.encoder.layers.5.self_attn.v_proj.bias', 'vision_model.encoder.layers.8.layer_norm2.bias', 'vision_model.encoder.layers.3.mlp.fc2.bias', 'vision_model.encoder.layers.16.layer_norm1.weight', 'vision_model.encoder.layers.21.layer_norm1.bias', 'vision_model.encoder.layers.6.layer_norm1.weight', 'vision_model.encoder.layers.6.self_attn.q_proj.bias', 'vision_model.encoder.layers.4.self_attn.out_proj.weight', 'vision_model.encoder.layers.7.layer_norm2.weight', 'vision_model.encoder.layers.9.mlp.fc1.bias', 'vision_model.encoder.layers.12.self_attn.k_proj.bias', 'vision_model.encoder.layers.19.self_attn.out_proj.weight', 'vision_model.encoder.layers.8.self_attn.out_proj.weight', 'vision_model.encoder.layers.19.layer_norm1.weight', 'vision_model.encoder.layers.17.layer_norm1.bias', 'vision_model.encoder.layers.21.layer_norm1.weight', 'vision_model.encoder.layers.7.layer_norm2.bias', 'vision_model.encoder.layers.23.self_attn.k_proj.weight', 'vision_model.encoder.layers.4.mlp.fc1.bias', 'vision_model.encoder.layers.3.self_attn.q_proj.bias', 'vision_model.encoder.layers.18.layer_norm2.bias', 'vision_model.encoder.layers.21.self_attn.v_proj.weight', 'vision_model.encoder.layers.17.mlp.fc1.bias', 'vision_model.encoder.layers.19.self_attn.v_proj.bias', 'vision_model.encoder.layers.12.self_attn.q_proj.weight', 'vision_model.encoder.layers.14.self_attn.v_proj.weight', 'vision_model.encoder.layers.5.self_attn.k_proj.weight', 'vision_model.encoder.layers.9.self_attn.q_proj.weight', 'vision_model.encoder.layers.2.mlp.fc1.weight', 'vision_model.encoder.layers.18.layer_norm1.bias', 'vision_model.encoder.layers.15.layer_norm1.bias', 'vision_model.encoder.layers.2.layer_norm2.bias', 'vision_model.encoder.layers.10.self_attn.v_proj.weight', 'vision_model.encoder.layers.7.self_attn.k_proj.weight', 'vision_model.encoder.layers.15.self_attn.v_proj.weight', 'vision_model.encoder.layers.22.layer_norm1.weight', 'vision_model.encoder.layers.22.layer_norm1.bias', 'vision_model.encoder.layers.3.self_attn.out_proj.bias', 'vision_model.encoder.layers.23.self_attn.v_proj.bias', 'vision_model.encoder.layers.15.layer_norm2.weight', 'vision_model.encoder.layers.15.mlp.fc1.bias', 'vision_model.encoder.layers.20.layer_norm2.weight', 'vision_model.encoder.layers.1.self_attn.v_proj.weight', 'vision_model.encoder.layers.0.layer_norm2.bias', 'vision_model.encoder.layers.17.layer_norm2.weight', 'vision_model.encoder.layers.8.mlp.fc1.weight', 'vision_model.encoder.layers.16.layer_norm1.bias', 'vision_model.encoder.layers.7.self_attn.out_proj.weight', 'vision_model.encoder.layers.19.layer_norm2.weight', 'vision_model.embeddings.position_embedding.weight', 'vision_model.encoder.layers.4.mlp.fc2.weight', 'vision_model.encoder.layers.1.layer_norm2.bias', 'vision_model.encoder.layers.14.mlp.fc2.bias', 'vision_model.encoder.layers.14.layer_norm1.weight', 'vision_model.encoder.layers.13.self_attn.out_proj.weight', 'vision_model.encoder.layers.12.self_attn.k_proj.weight', 'vision_model.encoder.layers.21.self_attn.v_proj.bias', 'vision_model.encoder.layers.6.mlp.fc1.bias', 'vision_model.encoder.layers.11.mlp.fc2.weight', 'vision_model.encoder.layers.8.self_attn.q_proj.weight', 'vision_model.encoder.layers.22.self_attn.k_proj.bias', 'vision_model.encoder.layers.22.self_attn.out_proj.weight', 'vision_model.encoder.layers.10.self_attn.k_proj.bias', 'vision_model.encoder.layers.22.layer_norm2.bias', 'vision_model.encoder.layers.17.self_attn.k_proj.weight', 'vision_model.encoder.layers.8.mlp.fc2.weight', 'vision_model.encoder.layers.1.layer_norm1.bias', 'vision_model.encoder.layers.4.layer_norm2.bias', 'vision_model.encoder.layers.19.layer_norm2.bias', 'vision_model.encoder.layers.16.self_attn.v_proj.bias', 'vision_model.encoder.layers.3.self_attn.k_proj.bias', 'vision_model.encoder.layers.23.layer_norm1.weight', 'vision_model.encoder.layers.6.mlp.fc2.weight', 'vision_model.encoder.layers.13.self_attn.q_proj.weight', 'vision_model.encoder.layers.23.self_attn.q_proj.weight', 'vision_model.encoder.layers.20.self_attn.out_proj.bias', 'vision_model.encoder.layers.13.layer_norm1.weight', 'vision_model.encoder.layers.16.self_attn.v_proj.weight', 'vision_model.encoder.layers.19.self_attn.k_proj.bias', 'vision_model.encoder.layers.5.layer_norm2.bias', 'vision_model.encoder.layers.9.self_attn.q_proj.bias', 'vision_model.encoder.layers.12.mlp.fc2.weight', 'vision_model.encoder.layers.8.layer_norm2.weight', 'vision_model.encoder.layers.10.layer_norm1.weight', 'vision_model.encoder.layers.16.self_attn.q_proj.weight', 'vision_model.encoder.layers.20.layer_norm1.weight', 'vision_model.encoder.layers.12.layer_norm2.weight', 'vision_model.encoder.layers.11.self_attn.q_proj.weight', 'vision_model.encoder.layers.13.self_attn.v_proj.weight', 'vision_model.encoder.layers.12.layer_norm1.bias', 'vision_model.encoder.layers.19.mlp.fc2.bias', 'vision_model.encoder.layers.0.layer_norm1.bias', 'vision_model.encoder.layers.18.self_attn.out_proj.bias', 'vision_model.encoder.layers.4.self_attn.q_proj.weight', 'vision_model.encoder.layers.13.self_attn.q_proj.bias', 'vision_model.encoder.layers.6.mlp.fc1.weight', 'vision_model.encoder.layers.22.mlp.fc1.bias', 'vision_model.encoder.layers.11.self_attn.out_proj.bias', 'vision_model.encoder.layers.18.self_attn.v_proj.bias', 'vision_model.encoder.layers.18.mlp.fc1.bias', 'vision_model.encoder.layers.16.mlp.fc2.weight', 'vision_model.encoder.layers.23.layer_norm1.bias', 'vision_model.embeddings.patch_embedding.weight', 'vision_model.encoder.layers.8.self_attn.q_proj.bias', 'vision_model.pre_layrnorm.weight', 'vision_model.encoder.layers.23.mlp.fc2.bias', 'vision_model.encoder.layers.15.self_attn.out_proj.bias', 'vision_model.encoder.layers.12.self_attn.q_proj.bias', 'vision_model.encoder.layers.21.self_attn.out_proj.bias', 'vision_model.encoder.layers.8.mlp.fc2.bias', 'vision_model.encoder.layers.9.self_attn.k_proj.weight', 'vision_model.encoder.layers.10.self_attn.q_proj.weight', 'vision_model.encoder.layers.14.mlp.fc1.bias', 'vision_model.encoder.layers.1.self_attn.q_proj.bias', 'text_projection.weight', 'vision_model.encoder.layers.18.self_attn.q_proj.bias', 'vision_model.encoder.layers.6.self_attn.k_proj.bias', 'vision_model.encoder.layers.13.mlp.fc1.weight', 'vision_model.encoder.layers.1.self_attn.out_proj.weight', 'vision_model.encoder.layers.14.mlp.fc2.weight', 'vision_model.encoder.layers.21.mlp.fc2.weight', 'vision_model.encoder.layers.1.mlp.fc1.bias', 'vision_model.encoder.layers.9.self_attn.k_proj.bias', 'vision_model.encoder.layers.4.self_attn.v_proj.weight', 'vision_model.encoder.layers.20.self_attn.k_proj.bias', 'vision_model.encoder.layers.19.layer_norm1.bias', 'vision_model.encoder.layers.2.mlp.fc2.weight', 'vision_model.encoder.layers.18.mlp.fc2.bias', 'vision_model.encoder.layers.13.layer_norm2.bias', 'vision_model.encoder.layers.14.mlp.fc1.weight', 'vision_model.encoder.layers.15.mlp.fc1.weight', 'vision_model.encoder.layers.10.mlp.fc1.bias', 'visual_projection.weight', 'vision_model.encoder.layers.4.mlp.fc2.bias', 'vision_model.encoder.layers.0.mlp.fc2.weight', 'vision_model.encoder.layers.0.mlp.fc2.bias', 'vision_model.encoder.layers.12.mlp.fc1.weight', 'vision_model.encoder.layers.17.layer_norm2.bias', 'vision_model.encoder.layers.20.self_attn.out_proj.weight', 'vision_model.encoder.layers.10.self_attn.out_proj.bias', 'vision_model.encoder.layers.17.layer_norm1.weight', 'vision_model.encoder.layers.2.mlp.fc2.bias', 'vision_model.encoder.layers.21.self_attn.q_proj.bias', 'vision_model.encoder.layers.2.self_attn.v_proj.bias', 'vision_model.encoder.layers.5.self_attn.q_proj.weight', 'vision_model.encoder.layers.3.self_attn.out_proj.weight', 'vision_model.encoder.layers.4.self_attn.out_proj.bias', 'vision_model.encoder.layers.23.mlp.fc1.weight', 'vision_model.encoder.layers.23.layer_norm2.bias', 'vision_model.encoder.layers.10.layer_norm2.weight', 'vision_model.encoder.layers.14.self_attn.out_proj.bias', 'vision_model.encoder.layers.7.self_attn.k_proj.bias', 'vision_model.encoder.layers.2.self_attn.v_proj.weight', 'vision_model.encoder.layers.5.self_attn.k_proj.bias', 'vision_model.encoder.layers.20.self_attn.v_proj.bias']
- This IS expected if you are initializing CLIPTextModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing CLIPTextModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Traceback (most recent call last):
  File "/content/convert_original_stable_diffusion_to_diffusers.py", line 687, in <module>
    text_model = convert_ldm_clip_checkpoint(checkpoint)
  File "/content/convert_original_stable_diffusion_to_diffusers.py", line 609, in convert_ldm_clip_checkpoint
    text_model.load_state_dict(text_model_dict)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1605, in load_state_dict
    self.__class__.__name__, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for CLIPTextModel:
	Missing key(s) in state_dict: "text_model.embeddings.position_ids", "text_model.embeddings.token_embedding.weight", "text_model.embeddings.position_embedding.weight", "text_model.encoder.layers.0.self_attn.k_proj.weight", "text_model.encoder.layers.0.self_attn.k_proj.bias", "text_model.encoder.layers.0.self_attn.v_proj.weight", "text_model.encoder.layers.0.self_attn.v_proj.bias", "text_model.encoder.layers.0.self_attn.q_proj.weight", "text_model.encoder.layers.0.self_attn.q_proj.bias", "text_model.encoder.layers.0.self_attn.out_proj.weight", "text_model.encoder.layers.0.self_attn.out_proj.bias", "text_model.encoder.layers.0.layer_norm1.weight", "text_model.encoder.layers.0.layer_norm1.bias", "text_model.encoder.layers.0.mlp.fc1.weight", "text_model.encoder.layers.0.mlp.fc1.bias", "text_model.encoder.layers.0.mlp.fc2.weight", "text_model.encoder.layers.0.mlp.fc2.bias", "text_model.encoder.layers.0.layer_norm2.weight", "text_model.encoder.layers.0.layer_norm2.bias", "text_model.encoder.layers.1.self_attn.k_proj.weight", "text_model.encoder.layers.1.self_attn.k_proj.bias", "text_model.encoder.layers.1.self_attn.v_proj.weight", "text_model.encoder.layers.1.self_attn.v_proj.bias", "text_model.encoder.layers.1.self_attn.q_proj.weight", "text_model.encoder.layers.1.self_attn.q_proj.bias", "text_model.encoder.layers.1.self_attn.out_proj.weight", "text_model.encoder.layers.1.self_attn.out_proj.bias", "text_model.encoder.layers.1.layer_norm1.weight", "text_model.encoder.layers.1.layer_norm1.bias", "text_model.encoder.layers.1.mlp.fc1.weight", "text_model.encoder.layers.1.mlp.fc1.bias", "text_model.encoder.layers.1.mlp.fc2.weight", "text_model.encoder.layers.1.mlp.fc2.bias", "text_model.encoder.layers.1.layer_norm2.weight", "text_model.encoder.layers.1.layer_norm2.bias", "text_model.encoder.layers.2.self_attn.k_proj.weight", "text_model.encoder.layers.2.self_attn.k_proj.bias", "text_model.encoder.layers.2.self_attn.v_proj.weight", "text_model.encoder.layers.2.self_attn.v_proj.bias", "text_model.encoder.layers.2.self_attn.q_proj.weight", "text_model.encoder.layers.2.self_attn.q_proj.bias", "text_model.encoder.layers.2.self_attn.out_proj.weight", "text_model.encoder.layers.2.self_attn.out_proj.bias", "text_model.encoder.layers.2.layer_norm1.weight", "text_model.encoder.layers.2.layer_norm1.bias", "text_model.encoder.layers.2.mlp.fc1.weight", "text_model.encoder.layers.2.mlp.fc1.bias", "text_model.encoder.layers.2.mlp.fc2.weight", "text_model.encoder.layers.2.mlp.fc2.bias", "text_model.encoder.layers.2.layer_norm2.weight", "text_model.encoder.layers.2.layer_norm2.bias", "text_model.encoder.layers.3.self_attn.k_proj.weight", "text_model.encoder.layers.3.self_attn.k_proj.bias", "text_model.encoder.layers.3.self_attn.v_proj.weight", "text_model.encoder.layers.3.self_attn.v_proj.bias", "text_model.encoder.layers.3.self_attn.q_proj.weight", "text_model.encoder.layers.3.self_attn.q_proj.bias", "text_model.encoder.layers.3.self_attn.out_proj.weight", "text_model.encoder.layers.3.self_attn.out_proj.bias", "text_model.encoder.layers.3.layer_norm1.weight", "text_model.encoder.layers.3.layer_norm1.bias", "text_model.encoder.layers.3.mlp.fc1.weight", "text_model.encoder.layers.3.mlp.fc1.bias", "text_model.encoder.layers.3.mlp.fc2.weight", "text_model.encoder.layers.3.mlp.fc2.bias", "text_model.encoder.layers.3.layer_norm2.weight", "text_model.encoder.layers.3.layer_norm2.bias", "text_model.encoder.layers.4.self_attn.k_proj.weight", "text_model.encoder.layers.4.self_attn.k_proj.bias", "text_model.encoder.layers.4.self_attn.v_proj.weight", "text_model.encoder.layers.4.self_attn.v_proj.bias", "text_model.encoder.layers.4.self_attn.q_proj.weight", "text_model.encoder.layers.4.self_attn.q_proj.bias", "text_model.encoder.layers.4.self_attn.out_proj.weight", "text_model.encoder.layers.4.self_attn.out_proj.bias", "text_model.encoder.layers.4.layer_norm1.weight", "text_model.encoder.layers.4.layer_norm1.bias", "text_model.encoder.layers.4.mlp.fc1.weight", "text_model.encoder.layers.4.mlp.fc1.bias", "text_model.encoder.layers.4.mlp.fc2.weight", "text_model.encoder.layers.4.mlp.fc2.bias", "text_model.encoder.layers.4.layer_norm2.weight", "text_model.encoder.layers.4.layer_norm2.bias", "text_model.encoder.layers.5.self_attn.k_proj.weight", "text_model.encoder.layers.5.self_attn.k_proj.bias", "text_model.encoder.layers.5.self_attn.v_proj.weight", "text_model.encoder.layers.5.self_attn.v_proj.bias", "text_model.encoder.layers.5.self_attn.q_proj.weight", "text_model.encoder.layers.5.self_attn.q_proj.bias", "text_model.encoder.layers.5.self_attn.out_proj.weight", "text_model.encoder.layers.5.self_attn.out_proj.bias", "text_model.encoder.layers.5.layer_norm1.weight", "text_model.encoder.layers.5.layer_norm1.bias", "text_model.encoder.layers.5.mlp.fc1.weight", "text_model.encoder.layers.5.mlp.fc1.bias", "text_model.encoder.layers.5.mlp.fc2.weight", "text_model.encoder.layers.5.mlp.fc2.bias", "text_model.encoder.layers.5.layer_norm2.weight", "text_model.encoder.layers.5.layer_norm2.bias", "text_model.encoder.layers.6.self_attn.k_proj.weight", "text_model.encoder.layers.6.self_attn.k_proj.bias", "text_model.encoder.layers.6.self_attn.v_proj.weight", "text_model.encoder.layers.6.self_attn.v_proj.bias", "text_model.encoder.layers.6.self_attn.q_proj.weight", "text_model.encoder.layers.6.self_attn.q_proj.bias", "text_model.encoder.layers.6.self_attn.out_proj.weight", "text_model.encoder.layers.6.self_attn.out_proj.bias", "text_model.encoder.layers.6.layer_norm1.weight", "text_model.encoder.layers.6.layer_norm1.bias", "text_model.encoder.layers.6.mlp.fc1.weight", "text_model.encoder.layers.6.mlp.fc1.bias", "text_model.encoder.layers.6.mlp.fc2.weight", "text_model.encoder.layers.6.mlp.fc2.bias", "text_model.encoder.layers.6.layer_norm2.weight", "text_model.encoder.layers.6.layer_norm2.bias", "text_model.encoder.layers.7.self_attn.k_proj.weight", "text_model.encoder.layers.7.self_attn.k_proj.bias", "text_model.encoder.layers.7.self_attn.v_proj.weight", "text_model.encoder.layers.7.self_attn.v_proj.bias", "text_model.encoder.layers.7.self_attn.q_proj.weight", "text_model.encoder.layers.7.self_attn.q_proj.bias", "text_model.encoder.layers.7.self_attn.out_proj.weight", "text_model.encoder.layers.7.self_attn.out_proj.bias", "text_model.encoder.layers.7.layer_norm1.weight", "text_model.encoder.layers.7.layer_norm1.bias", "text_model.encoder.layers.7.mlp.fc1.weight", "text_model.encoder.layers.7.mlp.fc1.bias", "text_model.encoder.layers.7.mlp.fc2.weight", "text_model.encoder.layers.7.mlp.fc2.bias", "text_model.encoder.layers.7.layer_norm2.weight", "text_model.encoder.layers.7.layer_norm2.bias", "text_model.encoder.layers.8.self_attn.k_proj.weight", "text_model.encoder.layers.8.self_attn.k_proj.bias", "text_model.encoder.layers.8.self_attn.v_proj.weight", "text_model.encoder.layers.8.self_attn.v_proj.bias", "text_model.encoder.layers.8.self_attn.q_proj.weight", "text_model.encoder.layers.8.self_attn.q_proj.bias", "text_model.encoder.layers.8.self_attn.out_proj.weight", "text_model.encoder.layers.8.self_attn.out_proj.bias", "text_model.encoder.layers.8.layer_norm1.weight", "text_model.encoder.layers.8.layer_norm1.bias", "text_model.encoder.layers.8.mlp.fc1.weight", "text_model.encoder.layers.8.mlp.fc1.bias", "text_model.encoder.layers.8.mlp.fc2.weight", "text_model.encoder.layers.8.mlp.fc2.bias", "text_model.encoder.layers.8.layer_norm2.weight", "text_model.encoder.layers.8.layer_norm2.bias", "text_model.encoder.layers.9.self_attn.k_proj.weight", "text_model.encoder.layers.9.self_attn.k_proj.bias", "text_model.encoder.layers.9.self_attn.v_proj.weight", "text_model.encoder.layers.9.self_attn.v_proj.bias", "text_model.encoder.layers.9.self_attn.q_proj.weight", "text_model.encoder.layers.9.self_attn.q_proj.bias", "text_model.encoder.layers.9.self_attn.out_proj.weight", "text_model.encoder.layers.9.self_attn.out_proj.bias", "text_model.encoder.layers.9.layer_norm1.weight", "text_model.encoder.layers.9.layer_norm1.bias", "text_model.encoder.layers.9.mlp.fc1.weight", "text_model.encoder.layers.9.mlp.fc1.bias", "text_model.encoder.layers.9.mlp.fc2.weight", "text_model.encoder.layers.9.mlp.fc2.bias", "text_model.encoder.layers.9.layer_norm2.weight", "text_model.encoder.layers.9.layer_norm2.bias", "text_model.encoder.layers.10.self_attn.k_proj.weight", "text_model.encoder.layers.10.self_attn.k_proj.bias", "text_model.encoder.layers.10.self_attn.v_proj.weight", "text_model.encoder.layers.10.self_attn.v_proj.bias", "text_model.encoder.layers.10.self_attn.q_proj.weight", "text_model.encoder.layers.10.self_attn.q_proj.bias", "text_model.encoder.layers.10.self_attn.out_proj.weight", "text_model.encoder.layers.10.self_attn.out_proj.bias", "text_model.encoder.layers.10.layer_norm1.weight", "text_model.encoder.layers.10.layer_norm1.bias", "text_model.encoder.layers.10.mlp.fc1.weight", "text_model.encoder.layers.10.mlp.fc1.bias", "text_model.encoder.layers.10.mlp.fc2.weight", "text_model.encoder.layers.10.mlp.fc2.bias", "text_model.encoder.layers.10.layer_norm2.weight", "text_model.encoder.layers.10.layer_norm2.bias", "text_model.encoder.layers.11.self_attn.k_proj.weight", "text_model.encoder.layers.11.self_attn.k_proj.bias", "text_model.encoder.layers.11.self_attn.v_proj.weight", "text_model.encoder.layers.11.self_attn.v_proj.bias", "text_model.encoder.layers.11.self_attn.q_proj.weight", "text_model.encoder.layers.11.self_attn.q_proj.bias", "text_model.encoder.layers.11.self_attn.out_proj.weight", "text_model.encoder.layers.11.self_attn.out_proj.bias", "text_model.encoder.layers.11.layer_norm1.weight", "text_model.encoder.layers.11.layer_norm1.bias", "text_model.encoder.layers.11.mlp.fc1.weight", "text_model.encoder.layers.11.mlp.fc1.bias", "text_model.encoder.layers.11.mlp.fc2.weight", "text_model.encoder.layers.11.mlp.fc2.bias", "text_model.encoder.layers.11.layer_norm2.weight", "text_model.encoder.layers.11.layer_norm2.bias", "text_model.final_layer_norm.weight", "text_model.final_layer_norm.bias". 
	Unexpected key(s) in state_dict: "embeddings.position_ids", "embeddings.token_embedding.weight", "embeddings.position_embedding.weight", "encoder.layers.0.self_attn.k_proj.weight", "encoder.layers.0.self_attn.k_proj.bias", "encoder.layers.0.self_attn.v_proj.weight", "encoder.layers.0.self_attn.v_proj.bias", "encoder.layers.0.self_attn.q_proj.weight", "encoder.layers.0.self_attn.q_proj.bias", "encoder.layers.0.self_attn.out_proj.weight", "encoder.layers.0.self_attn.out_proj.bias", "encoder.layers.0.layer_norm1.weight", "encoder.layers.0.layer_norm1.bias", "encoder.layers.0.mlp.fc1.weight", "encoder.layers.0.mlp.fc1.bias", "encoder.layers.0.mlp.fc2.weight", "encoder.layers.0.mlp.fc2.bias", "encoder.layers.0.layer_norm2.weight", "encoder.layers.0.layer_norm2.bias", "encoder.layers.1.self_attn.k_proj.weight", "encoder.layers.1.self_attn.k_proj.bias", "encoder.layers.1.self_attn.v_proj.weight", "encoder.layers.1.self_attn.v_proj.bias", "encoder.layers.1.self_attn.q_proj.weight", "encoder.layers.1.self_attn.q_proj.bias", "encoder.layers.1.self_attn.out_proj.weight", "encoder.layers.1.self_attn.out_proj.bias", "encoder.layers.1.layer_norm1.weight", "encoder.layers.1.layer_norm1.bias", "encoder.layers.1.mlp.fc1.weight", "encoder.layers.1.mlp.fc1.bias", "encoder.layers.1.mlp.fc2.weight", "encoder.layers.1.mlp.fc2.bias", "encoder.layers.1.layer_norm2.weight", "encoder.layers.1.layer_norm2.bias", "encoder.layers.2.self_attn.k_proj.weight", "encoder.layers.2.self_attn.k_proj.bias", "encoder.layers.2.self_attn.v_proj.weight", "encoder.layers.2.self_attn.v_proj.bias", "encoder.layers.2.self_attn.q_proj.weight", "encoder.layers.2.self_attn.q_proj.bias", "encoder.layers.2.self_attn.out_proj.weight", "encoder.layers.2.self_attn.out_proj.bias", "encoder.layers.2.layer_norm1.weight", "encoder.layers.2.layer_norm1.bias", "encoder.layers.2.mlp.fc1.weight", "encoder.layers.2.mlp.fc1.bias", "encoder.layers.2.mlp.fc2.weight", "encoder.layers.2.mlp.fc2.bias", "encoder.layers.2.layer_norm2.weight", "encoder.layers.2.layer_norm2.bias", "encoder.layers.3.self_attn.k_proj.weight", "encoder.layers.3.self_attn.k_proj.bias", "encoder.layers.3.self_attn.v_proj.weight", "encoder.layers.3.self_attn.v_proj.bias", "encoder.layers.3.self_attn.q_proj.weight", "encoder.layers.3.self_attn.q_proj.bias", "encoder.layers.3.self_attn.out_proj.weight", "encoder.layers.3.self_attn.out_proj.bias", "encoder.layers.3.layer_norm1.weight", "encoder.layers.3.layer_norm1.bias", "encoder.layers.3.mlp.fc1.weight", "encoder.layers.3.mlp.fc1.bias", "encoder.layers.3.mlp.fc2.weight", "encoder.layers.3.mlp.fc2.bias", "encoder.layers.3.layer_norm2.weight", "encoder.layers.3.layer_norm2.bias", "encoder.layers.4.self_attn.k_proj.weight", "encoder.layers.4.self_attn.k_proj.bias", "encoder.layers.4.self_attn.v_proj.weight", "encoder.layers.4.self_attn.v_proj.bias", "encoder.layers.4.self_attn.q_proj.weight", "encoder.layers.4.self_attn.q_proj.bias", "encoder.layers.4.self_attn.out_proj.weight", "encoder.layers.4.self_attn.out_proj.bias", "encoder.layers.4.layer_norm1.weight", "encoder.layers.4.layer_norm1.bias", "encoder.layers.4.mlp.fc1.weight", "encoder.layers.4.mlp.fc1.bias", "encoder.layers.4.mlp.fc2.weight", "encoder.layers.4.mlp.fc2.bias", "encoder.layers.4.layer_norm2.weight", "encoder.layers.4.layer_norm2.bias", "encoder.layers.5.self_attn.k_proj.weight", "encoder.layers.5.self_attn.k_proj.bias", "encoder.layers.5.self_attn.v_proj.weight", "encoder.layers.5.self_attn.v_proj.bias", "encoder.layers.5.self_attn.q_proj.weight", "encoder.layers.5.self_attn.q_proj.bias", "encoder.layers.5.self_attn.out_proj.weight", "encoder.layers.5.self_attn.out_proj.bias", "encoder.layers.5.layer_norm1.weight", "encoder.layers.5.layer_norm1.bias", "encoder.layers.5.mlp.fc1.weight", "encoder.layers.5.mlp.fc1.bias", "encoder.layers.5.mlp.fc2.weight", "encoder.layers.5.mlp.fc2.bias", "encoder.layers.5.layer_norm2.weight", "encoder.layers.5.layer_norm2.bias", "encoder.layers.6.self_attn.k_proj.weight", "encoder.layers.6.self_attn.k_proj.bias", "encoder.layers.6.self_attn.v_proj.weight", "encoder.layers.6.self_attn.v_proj.bias", "encoder.layers.6.self_attn.q_proj.weight", "encoder.layers.6.self_attn.q_proj.bias", "encoder.layers.6.self_attn.out_proj.weight", "encoder.layers.6.self_attn.out_proj.bias", "encoder.layers.6.layer_norm1.weight", "encoder.layers.6.layer_norm1.bias", "encoder.layers.6.mlp.fc1.weight", "encoder.layers.6.mlp.fc1.bias", "encoder.layers.6.mlp.fc2.weight", "encoder.layers.6.mlp.fc2.bias", "encoder.layers.6.layer_norm2.weight", "encoder.layers.6.layer_norm2.bias", "encoder.layers.7.self_attn.k_proj.weight", "encoder.layers.7.self_attn.k_proj.bias", "encoder.layers.7.self_attn.v_proj.weight", "encoder.layers.7.self_attn.v_proj.bias", "encoder.layers.7.self_attn.q_proj.weight", "encoder.layers.7.self_attn.q_proj.bias", "encoder.layers.7.self_attn.out_proj.weight", "encoder.layers.7.self_attn.out_proj.bias", "encoder.layers.7.layer_norm1.weight", "encoder.layers.7.layer_norm1.bias", "encoder.layers.7.mlp.fc1.weight", "encoder.layers.7.mlp.fc1.bias", "encoder.layers.7.mlp.fc2.weight", "encoder.layers.7.mlp.fc2.bias", "encoder.layers.7.layer_norm2.weight", "encoder.layers.7.layer_norm2.bias", "encoder.layers.8.self_attn.k_proj.weight", "encoder.layers.8.self_attn.k_proj.bias", "encoder.layers.8.self_attn.v_proj.weight", "encoder.layers.8.self_attn.v_proj.bias", "encoder.layers.8.self_attn.q_proj.weight", "encoder.layers.8.self_attn.q_proj.bias", "encoder.layers.8.self_attn.out_proj.weight", "encoder.layers.8.self_attn.out_proj.bias", "encoder.layers.8.layer_norm1.weight", "encoder.layers.8.layer_norm1.bias", "encoder.layers.8.mlp.fc1.weight", "encoder.layers.8.mlp.fc1.bias", "encoder.layers.8.mlp.fc2.weight", "encoder.layers.8.mlp.fc2.bias", "encoder.layers.8.layer_norm2.weight", "encoder.layers.8.layer_norm2.bias", "encoder.layers.9.self_attn.k_proj.weight", "encoder.layers.9.self_attn.k_proj.bias", "encoder.layers.9.self_attn.v_proj.weight", "encoder.layers.9.self_attn.v_proj.bias", "encoder.layers.9.self_attn.q_proj.weight", "encoder.layers.9.self_attn.q_proj.bias", "encoder.layers.9.self_attn.out_proj.weight", "encoder.layers.9.self_attn.out_proj.bias", "encoder.layers.9.layer_norm1.weight", "encoder.layers.9.layer_norm1.bias", "encoder.layers.9.mlp.fc1.weight", "encoder.layers.9.mlp.fc1.bias", "encoder.layers.9.mlp.fc2.weight", "encoder.layers.9.mlp.fc2.bias", "encoder.layers.9.layer_norm2.weight", "encoder.layers.9.layer_norm2.bias", "encoder.layers.10.self_attn.k_proj.weight", "encoder.layers.10.self_attn.k_proj.bias", "encoder.layers.10.self_attn.v_proj.weight", "encoder.layers.10.self_attn.v_proj.bias", "encoder.layers.10.self_attn.q_proj.weight", "encoder.layers.10.self_attn.q_proj.bias", "encoder.layers.10.self_attn.out_proj.weight", "encoder.layers.10.self_attn.out_proj.bias", "encoder.layers.10.layer_norm1.weight", "encoder.layers.10.layer_norm1.bias", "encoder.layers.10.mlp.fc1.weight", "encoder.layers.10.mlp.fc1.bias", "encoder.layers.10.mlp.fc2.weight", "encoder.layers.10.mlp.fc2.bias", "encoder.layers.10.layer_norm2.weight", "encoder.layers.10.layer_norm2.bias", "encoder.layers.11.self_attn.k_proj.weight", "encoder.layers.11.self_attn.k_proj.bias", "encoder.layers.11.self_attn.v_proj.weight", "encoder.layers.11.self_attn.v_proj.bias", "encoder.layers.11.self_attn.q_proj.weight", "encoder.layers.11.self_attn.q_proj.bias", "encoder.layers.11.self_attn.out_proj.weight", "encoder.layers.11.self_attn.out_proj.bias", "encoder.layers.11.layer_norm1.weight", "encoder.layers.11.layer_norm1.bias", "encoder.layers.11.mlp.fc1.weight", "encoder.layers.11.mlp.fc1.bias", "encoder.layers.11.mlp.fc2.weight", "encoder.layers.11.mlp.fc2.bias", "encoder.layers.11.layer_norm2.weight", "encoder.layers.11.layer_norm2.bias", "final_layer_norm.weight", "final_layer_norm.bias". 
Conversion error, check your CKPT and try again
Conversion error, check your CKPT and try again

[TheLastBen]

it seems that NAI uses a different architecture for their CKPTs

[toyxyz]

I also had the same problem. I solved the problem by first converting the Nai to Diffuser using the old version convert_original_stable_diffusion_to_diffusers.py and back to Ckpt using convert_diffusers_to_original_stable_diffusion.py No more errors and training proceeds normally.

https://github.com/huggingface/diffusers/blob/039958eae55ff0700cfb42a7e72739575ab341f1/scripts/convert_original_stable_diffusion_to_diffusers.py

convert_original_stable_diffusion_to_diffusers.py

python convert_original_stable_diffusion_to_diffusers.py --checkpoint_path C:\ckpt\nai.ckpt --dump_path C:\ckpt\model --original_config_file C:\ckpt\v1-inference.yaml

convert_diffusers_to_original_stable_diffusion.py

python convert_diffusers_to_original_stable_diffusion.py --model_path C:\ckpt\model --checkpoint_path C:\ckpt\nai.ckpt

[toyxyz]

Oh thanks for letting me know! Can you tell me what effect it has? As a result of the test, there was no difference between the images generated by the modified nai ckpt and the original nai ckpt.

[TheLastBen]

if there is no difference there is no problem

[wktra]

@toyxyz this is in reference to the Nai 7gb full-latest model, right? I haven't been able to use that 7GB version or train using the anything3.0 7gb ckpt.

Will I be able to use these to train in this colab by following your advice?

For now, I've only been able to train using the 4GB versions of Nai and Anything 3.0

@TheLastBen I had no idea we can talk about these models on this forum. I thought you'd be against bringing them up. Everyone else is.

[TheLastBen]

We can discuss the general issues, like RAM issues caused by 7GB ckpts or incompatible ckpts

[wktra]

We can discuss the general issues, like RAM issues caused by 7GB ckpts or incompatible ckpts

I pretty much almost gave up trying to use the full versions of NAI and Anything3.0, but I'm going to try toyxys's tricks and see if it works...

[wktra]

@toyxyz omg this is brutal 😥
I'm trying to do this locally with very little knowledge of python. Keep getting GPU errors about CUDA. Please tell me there's an easier way to do this via colab where I can just link to the model in gdrive and click a button to convert...

I'm adamant that I want to use the 7GB NAi and Anything3.0 ckpts to train using this colab!

[wktra]

Got some errors, but it seems Nai converted back and forth just fine.
anything3.0.ckpt gave this error: KeyError: 'time_embed.0.weight'

sigh