Segmentation fault

[daxime]

Hi,
Me again. I have a segmentation fault issue when I try to generate a video.

./sd -M vid_gen --diffusion-model ./models/wan/Wan2.2-TI2V-5B-Q4_0.gguf \
    --vae ./models/wan/wan2.2_vae.safetensors \
    --t5xxl ./models/wan/umt5-xxl-encoder-Q3_K_S.gguf \
    -p "a lovely girl" \
    --cfg-scale 6.0 \
    --sampling-method euler -v -W 480 -H 832 \
    --diffusion-fa \
    --offload-to-cpu \
    --video-frames 33 \
    --flow-shift 3.0

Option:
    n_threads:                         6
    mode:                              vid_gen
    model_path:
    wtype:                             unspecified
    clip_l_path:
    clip_g_path:
    clip_vision_path:
    t5xxl_path:                        ./models/wan/umt5-xxl-encoder-Q3_K_S.gguf
    diffusion_model_path:              ./models/wan/Wan2.2-TI2V-5B-Q4_0.gguf
    high_noise_diffusion_model_path:
    vae_path:                          ./models/wan/wan2.2_vae.safetensors
    taesd_path:
    esrgan_path:
    control_net_path:
    embedding_dir:
    stacked_id_embed_dir:
    input_id_images_path:
    style ratio:                       20.00
    normalize input image:             false
    output_path:                       output.png
    init_image_path:
    end_image_path:
    mask_image_path:
    control_image_path:
    ref_images_paths:
    increase_ref_index:                false
    offload_params_to_cpu:             true
    clip_on_cpu:                       false
    control_net_cpu:                   false
    vae_on_cpu:                        false
    diffusion flash attention:         true
    diffusion Conv2d direct:           false
    vae_conv_direct:                   false
    control_strength:                  0.90
    prompt:                            a lovely girl
    negative_prompt:
    clip_skip:                         -1
    width:                             480
    height:                            832
    sample_params:                     (txt_cfg: 6.00, img_cfg: 6.00, distilled_guidance: 3.50, slg.layer_count: 3, slg.layer_start: 0.01, slg.layer_end: 0.20, slg.scale: 0.00, scheduler: default, sample_method: euler, sample_steps: 20, eta: 0.00)
    high_noise_sample_params:          (txt_cfg: 7.00, img_cfg: 7.00, distilled_guidance: 3.50, slg.layer_count: 3, slg.layer_start: 0.01, slg.layer_end: 0.20, slg.scale: 0.00, scheduler: default, sample_method: euler_a, sample_steps: -1, eta: 0.00)
    moe_boundary:                      0.875
    flow_shift:                        3.00
    strength(img2img):                 0.75
    rng:                               cuda
    seed:                              42
    batch_count:                       1
    vae_tiling:                        false
    upscale_repeats:                   1
    chroma_use_dit_mask:               true
    chroma_use_t5_mask:                false
    chroma_t5_mask_pad:                1
    video_frames:                      33
    fps:                               16
System Info:
    SSE3 = 1
    AVX = 1
    AVX2 = 1
    AVX512 = 0
    AVX512_VBMI = 0
    AVX512_VNNI = 0
    FMA = 1
    NEON = 0
    ARM_FMA = 0
    F16C = 1
    FP16_VA = 0
    WASM_SIMD = 0
    VSX = 0
[DEBUG] stable-diffusion.cpp:151  - Using Vulkan backend
[DEBUG] ggml_extend.hpp:62   - ggml_vulkan: Found 1 Vulkan devices:
[DEBUG] ggml_extend.hpp:62   - ggml_vulkan: 0 = Radeon RX 5500 XT (RADV NAVI14) (radv) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 65536 | int dot: 0 | matrix cores: none
[INFO ] stable-diffusion.cpp:208  - loading diffusion model from './models/wan/Wan2.2-TI2V-5B-Q4_0.gguf'
[INFO ] model.cpp:1035 - load ./models/wan/Wan2.2-TI2V-5B-Q4_0.gguf using gguf format
[DEBUG] model.cpp:1052 - init from './models/wan/Wan2.2-TI2V-5B-Q4_0.gguf'
[ERROR] ggml_extend.hpp:71   - gguf_init_from_file_impl: tensor 'patch_embedding.weight' has invalid number of dimensions: 5 > 4
[ERROR] ggml_extend.hpp:71   - gguf_init_from_file_impl: failed to read tensor info
[ERROR] model.cpp:1061 - failed to open './models/wan/Wan2.2-TI2V-5B-Q4_0.gguf' with gguf_init_from_file. Try to open it with GGUFReader.
[DEBUG] gguf_reader.hpp:197  - GGUF v3, tensor_count=825, metadata_kv_count=3
[INFO ] stable-diffusion.cpp:248  - loading t5xxl from './models/wan/umt5-xxl-encoder-Q3_K_S.gguf'
[INFO ] model.cpp:1035 - load ./models/wan/umt5-xxl-encoder-Q3_K_S.gguf using gguf format
[DEBUG] model.cpp:1052 - init from './models/wan/umt5-xxl-encoder-Q3_K_S.gguf'
[INFO ] stable-diffusion.cpp:255  - loading vae from './models/wan/wan2.2_vae.safetensors'
[INFO ] model.cpp:1038 - load ./models/wan/wan2.2_vae.safetensors using safetensors format
[DEBUG] model.cpp:1145 - init from './models/wan/wan2.2_vae.safetensors', prefix = 'vae.'
[DEBUG] model.cpp:1778 - patch_embedding_channels 147456
[INFO ] stable-diffusion.cpp:267  - Version: Wan 2.2 TI2V
[INFO ] stable-diffusion.cpp:301  - Weight type:                 q4_0
[INFO ] stable-diffusion.cpp:302  - Conditioner weight type:     q3_K
[INFO ] stable-diffusion.cpp:303  - Diffusion model weight type: q4_0
[INFO ] stable-diffusion.cpp:304  - VAE weight type:             NONE
[DEBUG] stable-diffusion.cpp:306  - ggml tensor size = 400 bytes
[WARN ] stable-diffusion.cpp:335  - !!!It appears that you are using the T5 model. Some backends may encounter issues with it.If you notice that the generated images are completely black,try running the T5 model on the CPU using the --clip-on-cpu parameter.
[INFO ] stable-diffusion.cpp:345  - Using flash attention in the diffusion model
[INFO ] wan.hpp:1999 - Wan2.2-TI2V-5B
[DEBUG] ggml_extend.hpp:1609 - t5 params backend buffer size =  2720.00 MB(RAM) (242 tensors)
[DEBUG] ggml_extend.hpp:1609 - Wan2.2-TI2V-5B params backend buffer size =  2892.18 MB(RAM) (825 tensors)
[DEBUG] ggml_extend.hpp:1609 - wan_vae params backend buffer size =  1344.24 MB(RAM) (196 tensors)
[DEBUG] stable-diffusion.cpp:566  - loading weights
[DEBUG] model.cpp:1998 - loading tensors from ./models/wan/Wan2.2-TI2V-5B-Q4_0.gguf
  |==================================================| 1263/1263 - 58.95it/s
[DEBUG] model.cpp:1998 - loading tensors from ./models/wan/umt5-xxl-encoder-Q3_K_S.gguf
  |==================================================| 1263/1263 - 63.36it/s
[DEBUG] model.cpp:1998 - loading tensors from ./models/wan/wan2.2_vae.safetensors
  |==================================================| 1263/1263 - 176.05it/s
[INFO ] model.cpp:2216 - loading tensors completed, taking 48.64s (process: 0.10s, read: 48.50s, memcpy: 0.00s, convert: 0.03s, copy_to_backend: 0.00s)
[INFO ] stable-diffusion.cpp:647  - total params memory size = 6956.42MB (VRAM 6956.42MB, RAM 0.00MB): text_encoders 2720.00MB(VRAM), diffusion_model 2892.18MB(VRAM), vae 1344.24MB(VRAM), controlnet 0.00MB(VRAM), pmid 0.00MB(VRAM)
[INFO ] stable-diffusion.cpp:702  - running in FLOW mode
[DEBUG] stable-diffusion.cpp:726  - finished loaded file
[INFO ] stable-diffusion.cpp:2405 - generate_video 480x832x33
[INFO ] stable-diffusion.cpp:860  - attempting to apply 0 LoRAs
[INFO ] stable-diffusion.cpp:880  - apply_loras completed, taking 0.00s
[DEBUG] stable-diffusion.cpp:881  - prompt after extract and remove lora: "a lovely girl"
[DEBUG] conditioner.hpp:1287 - parse 'a lovely girl' to [['a lovely girl', 1], ]
[DEBUG] t5.hpp:402  - token length: 512
[INFO ] ggml_extend.hpp:1534 - t5 offload params (2720.00 MB, 242 tensors) to runtime backend (Vulkan0), taking 0.99s
[DEBUG] ggml_extend.hpp:1437 - t5 compute buffer size: 297.00 MB(VRAM)
[1]    10483 segmentation fault (core dumped)  ./sd -M vid_gen --diffusion-model ./models/wan/Wan2.2-TI2V-5B-Q4_0.gguf --vae

[Green-Sky]

Try with --clip-on-cpu

[daxime]

Now i have IOT instruction error:

Option:
    n_threads:                         6
    mode:                              vid_gen
    model_path:
    wtype:                             unspecified
    clip_l_path:
    clip_g_path:
    clip_vision_path:
    t5xxl_path:                        ./models/wan/umt5-xxl-encoder-Q3_K_S.gguf
    diffusion_model_path:              ./models/wan/Wan2.2-TI2V-5B-Q4_0.gguf
    high_noise_diffusion_model_path:
    vae_path:                          ./models/wan/wan2.2_vae.safetensors
    taesd_path:
    esrgan_path:
    control_net_path:
    embedding_dir:
    stacked_id_embed_dir:
    input_id_images_path:
    style ratio:                       20.00
    normalize input image:             false
    output_path:                       output.png
    init_image_path:
    end_image_path:
    mask_image_path:
    control_image_path:
    ref_images_paths:
    increase_ref_index:                false
    offload_params_to_cpu:             true
    clip_on_cpu:                       true
    control_net_cpu:                   false
    vae_on_cpu:                        false
    diffusion flash attention:         true
    diffusion Conv2d direct:           false
    vae_conv_direct:                   false
    control_strength:                  0.90
    prompt:                            a lovely girl
    negative_prompt:
    clip_skip:                         -1
    width:                             480
    height:                            832
    sample_params:                     (txt_cfg: 6.00, img_cfg: 6.00, distilled_guidance: 3.50, slg.layer_count: 3, slg.layer_start: 0.01, slg.layer_end: 0.20, slg.scale: 0.00, scheduler: default, sample_method: euler, sample_steps: 20, eta: 0.00)
    high_noise_sample_params:          (txt_cfg: 7.00, img_cfg: 7.00, distilled_guidance: 3.50, slg.layer_count: 3, slg.layer_start: 0.01, slg.layer_end: 0.20, slg.scale: 0.00, scheduler: default, sample_method: euler_a, sample_steps: -1, eta: 0.00)
    moe_boundary:                      0.875
    flow_shift:                        3.00
    strength(img2img):                 0.75
    rng:                               cuda
    seed:                              42
    batch_count:                       1
    vae_tiling:                        false
    upscale_repeats:                   1
    chroma_use_dit_mask:               true
    chroma_use_t5_mask:                false
    chroma_t5_mask_pad:                1
    video_frames:                      33
    fps:                               16
System Info:
    SSE3 = 1
    AVX = 1
    AVX2 = 1
    AVX512 = 0
    AVX512_VBMI = 0
    AVX512_VNNI = 0
    FMA = 1
    NEON = 0
    ARM_FMA = 0
    F16C = 1
    FP16_VA = 0
    WASM_SIMD = 0
    VSX = 0
[DEBUG] stable-diffusion.cpp:151  - Using Vulkan backend
[DEBUG] ggml_extend.hpp:62   - ggml_vulkan: Found 1 Vulkan devices:
[DEBUG] ggml_extend.hpp:62   - ggml_vulkan: 0 = Radeon RX 5500 XT (RADV NAVI14) (radv) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 65536 | int dot: 0 | matrix cores: none
[INFO ] stable-diffusion.cpp:208  - loading diffusion model from './models/wan/Wan2.2-TI2V-5B-Q4_0.gguf'
[INFO ] model.cpp:1035 - load ./models/wan/Wan2.2-TI2V-5B-Q4_0.gguf using gguf format
[DEBUG] model.cpp:1052 - init from './models/wan/Wan2.2-TI2V-5B-Q4_0.gguf'
[ERROR] ggml_extend.hpp:71   - gguf_init_from_file_impl: tensor 'patch_embedding.weight' has invalid number of dimensions: 5 > 4
[ERROR] ggml_extend.hpp:71   - gguf_init_from_file_impl: failed to read tensor info
[ERROR] model.cpp:1061 - failed to open './models/wan/Wan2.2-TI2V-5B-Q4_0.gguf' with gguf_init_from_file. Try to open it with GGUFReader.
[DEBUG] gguf_reader.hpp:197  - GGUF v3, tensor_count=825, metadata_kv_count=3
[INFO ] stable-diffusion.cpp:248  - loading t5xxl from './models/wan/umt5-xxl-encoder-Q3_K_S.gguf'
[INFO ] model.cpp:1035 - load ./models/wan/umt5-xxl-encoder-Q3_K_S.gguf using gguf format
[DEBUG] model.cpp:1052 - init from './models/wan/umt5-xxl-encoder-Q3_K_S.gguf'
[INFO ] stable-diffusion.cpp:255  - loading vae from './models/wan/wan2.2_vae.safetensors'
[INFO ] model.cpp:1038 - load ./models/wan/wan2.2_vae.safetensors using safetensors format
[DEBUG] model.cpp:1145 - init from './models/wan/wan2.2_vae.safetensors', prefix = 'vae.'
[DEBUG] model.cpp:1778 - patch_embedding_channels 147456
[INFO ] stable-diffusion.cpp:267  - Version: Wan 2.2 TI2V
[INFO ] stable-diffusion.cpp:301  - Weight type:                 q4_0
[INFO ] stable-diffusion.cpp:302  - Conditioner weight type:     q3_K
[INFO ] stable-diffusion.cpp:303  - Diffusion model weight type: q4_0
[INFO ] stable-diffusion.cpp:304  - VAE weight type:             NONE
[DEBUG] stable-diffusion.cpp:306  - ggml tensor size = 400 bytes
[INFO ] stable-diffusion.cpp:341  - CLIP: Using CPU backend
[INFO ] stable-diffusion.cpp:345  - Using flash attention in the diffusion model
[INFO ] wan.hpp:1999 - Wan2.2-TI2V-5B
[DEBUG] ggml_extend.hpp:1609 - t5 params backend buffer size =  2720.00 MB(RAM) (242 tensors)
[DEBUG] ggml_extend.hpp:1609 - Wan2.2-TI2V-5B params backend buffer size =  2892.18 MB(RAM) (825 tensors)
[DEBUG] ggml_extend.hpp:1609 - wan_vae params backend buffer size =  1344.24 MB(RAM) (196 tensors)
[DEBUG] stable-diffusion.cpp:566  - loading weights
[DEBUG] model.cpp:1998 - loading tensors from ./models/wan/Wan2.2-TI2V-5B-Q4_0.gguf
  |==================================================| 1263/1263 - 58.99it/s
[DEBUG] model.cpp:1998 - loading tensors from ./models/wan/umt5-xxl-encoder-Q3_K_S.gguf
  |==================================================| 1263/1263 - 63.43it/s
[DEBUG] model.cpp:1998 - loading tensors from ./models/wan/wan2.2_vae.safetensors
  |==================================================| 1263/1263 - 173.90it/s
[INFO ] model.cpp:2216 - loading tensors completed, taking 48.69s (process: 0.11s, read: 48.53s, memcpy: 0.00s, convert: 0.03s, copy_to_backend: 0.00s)
[INFO ] stable-diffusion.cpp:647  - total params memory size = 6956.42MB (VRAM 4236.42MB, RAM 2720.00MB): text_encoders 2720.00MB(RAM), diffusion_model 2892.18MB(VRAM), vae 1344.24MB(VRAM), controlnet 0.00MB(VRAM), pmid 0.00MB(RAM)
[INFO ] stable-diffusion.cpp:702  - running in FLOW mode
[DEBUG] stable-diffusion.cpp:726  - finished loaded file
[INFO ] stable-diffusion.cpp:2405 - generate_video 480x832x33
[INFO ] stable-diffusion.cpp:860  - attempting to apply 0 LoRAs
[INFO ] stable-diffusion.cpp:880  - apply_loras completed, taking 0.00s
[DEBUG] stable-diffusion.cpp:881  - prompt after extract and remove lora: "a lovely girl"
[DEBUG] conditioner.hpp:1287 - parse 'a lovely girl' to [['a lovely girl', 1], ]
[DEBUG] t5.hpp:402  - token length: 512
[DEBUG] ggml_extend.hpp:1437 - t5 compute buffer size: 297.00 MB(RAM)
[DEBUG] conditioner.hpp:1379 - computing condition graph completed, taking 40433 ms
[DEBUG] conditioner.hpp:1287 - parse '' to [['', 1], ]
[DEBUG] t5.hpp:402  - token length: 512
[DEBUG] ggml_extend.hpp:1437 - t5 compute buffer size: 297.00 MB(RAM)
[DEBUG] conditioner.hpp:1379 - computing condition graph completed, taking 39612 ms
[INFO ] stable-diffusion.cpp:2607 - get_learned_condition completed, taking 80067 ms
[DEBUG] stable-diffusion.cpp:2665 - sample 30x52x9
[INFO ] ggml_extend.hpp:1534 - Wan2.2-TI2V-5B offload params (2892.18 MB, 825 tensors) to runtime backend (Vulkan0), taking 1.18s
[DEBUG] ggml_extend.hpp:1437 - Wan2.2-TI2V-5B compute buffer size: 335.35 MB(VRAM)
ggml_vulkan: Error: Missing op: IM2COL_3D
/mnt/Sauvegardes/AI/AI_Diffusion/stable-diffusion.cpp/ggml/src/ggml-vulkan/ggml-vulkan.cpp:10308: fatal error
[New LWP 312237]
[New LWP 312236]
[New LWP 312235]
[New LWP 312234]
[New LWP 312233]
[New LWP 312013]
[New LWP 312012]
warning: could not find '.gnu_debugaltlink' file for /usr/lib/x86_64-linux-gnu/libvulkan_gfxstream.so
warning: could not find '.gnu_debugaltlink' file for /usr/lib/x86_64-linux-gnu/libvulkan_radeon.so
warning: could not find '.gnu_debugaltlink' file for /lib/x86_64-linux-gnu/libtinfo.so.6
warning: could not find '.gnu_debugaltlink' file for /usr/lib/x86_64-linux-gnu/libvulkan_intel_hasvk.so
warning: could not find '.gnu_debugaltlink' file for /usr/lib/x86_64-linux-gnu/libvulkan_intel.so
warning: could not find '.gnu_debugaltlink' file for /usr/lib/x86_64-linux-gnu/libvulkan_virtio.so
warning: could not find '.gnu_debugaltlink' file for /usr/lib/x86_64-linux-gnu/libvulkan_lvp.so
warning: could not find '.gnu_debugaltlink' file for /usr/lib/x86_64-linux-gnu/libvulkan_nouveau.so
warning: could not find '.gnu_debugaltlink' file for /lib/x86_64-linux-gnu/libVkLayer_MESA_device_select.so
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007f85567107e3 in __GI___wait4 (pid=312677, stat_loc=0x0, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
warning: 30     ../sysdeps/unix/sysv/linux/wait4.c: No such file or directory
#0  0x00007f85567107e3 in __GI___wait4 (pid=312677, stat_loc=0x0, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
30      in ../sysdeps/unix/sysv/linux/wait4.c
#1  0x000056273028c553 in ggml_print_backtrace ()
#2  0x000056273028c6f7 in ggml_abort ()
#3  0x0000562730267f9d in ggml_vk_build_graph(ggml_backend_vk_context*, ggml_cgraph*, int, ggml_tensor*, int, bool, bool, bool, bool) ()
#4  0x0000562730268363 in ggml_backend_vk_graph_compute(ggml_backend*, ggml_cgraph*) ()
#5  0x00005627302a4755 in ggml_backend_graph_compute ()
#6  0x000056273007185b in GGMLRunner::compute(std::function<ggml_cgraph* ()>, int, bool, ggml_tensor**, ggml_context*) ()
#7  0x0000562730072b76 in WanModel::compute(int, ggml_tensor*, ggml_tensor*, ggml_tensor*, ggml_tensor*, ggml_tensor*, ggml_tensor*, std::vector<ggml_tensor*, std::allocator<ggml_tensor*> >, bool, int, std::vector<ggml_tensor*, std::allocator<ggml_tensor*> >, float, ggml_tensor**, ggml_context*, std::vector<int, std::allocator<int> >) ()
#8  0x000056273007735a in StableDiffusionGGML::sample(ggml_context*, std::shared_ptr<DiffusionModel>, bool, ggml_tensor*, ggml_tensor*, SDCondition, SDCondition, SDCondition, ggml_tensor*, float, sd_guidance_params_t, float, sample_method_t, std::vector<float, std::allocator<float> > const&, int, SDCondition, std::vector<ggml_tensor*, std::allocator<ggml_tensor*> >, bool, ggml_tensor*)::{lambda(ggml_tensor*, float, int)#1}::operator()(ggml_tensor*, float, int) const ()
#9  0x000056273007cad5 in StableDiffusionGGML::sample(ggml_context*, std::shared_ptr<DiffusionModel>, bool, ggml_tensor*, ggml_tensor*, SDCondition, SDCondition, SDCondition, ggml_tensor*, float, sd_guidance_params_t, float, sample_method_t, std::vector<float, std::allocator<float> > const&, int, SDCondition, std::vector<ggml_tensor*, std::allocator<ggml_tensor*> >, bool, ggml_tensor*) ()
#10 0x000056273005be5f in generate_video ()
#11 0x000056272ffa8129 in main ()
[Inferior 1 (process 312004) detached]
[1]    312004 IOT instruction (core dumped)  ./sd -M vid_gen --diffusion-model ./models/wan/Wan2.2-TI2V-5B-Q4_0.gguf --vae

[Green-Sky]

ggml_vulkan: Error: Missing op: IM2COL_3D Right, ggml is currently missing this op.

Once ggml is synced from llama.cpp, to include ggml-org/llama.cpp#15795 , wan might work.
Also ggml-org/llama.cpp#15794.

[daxime]

I have seen your link, but not sure what I am supposed to do. I just have to wait for an update?

[Green-Sky]

Either you somehow apply those patches to the ggml in sd.cpp and compile it yourself, or yea, wait for an update.

[daxime]

Ok, thanks. I will wait :)