Whisper crash

[viktorprogger]

When I start Whisper on CUDA, I see such error. I tried .mp4, .mkv and .m4a files, result is the same. My graphics card is Radeon RX 7600, I already installed ROCm version 6.2 and installed pytorch with pip3 install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/rocm6.2. I work on Ubuntu 24.04.
Any help will be very appreciated. I have to transcribe a lot of hours of audio/video in the nearest future.

HSA_OVERRIDE_GFX_VERSION=11.0.2 doesn't change anything (I have gfx1102).

Traceback (most recent call last):
  File "/usr/local/bin/whisper", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/usr/local/lib/python3.12/dist-packages/whisper/transcribe.py", line 577, in cli
    model = load_model(model_name, device=device, download_root=model_dir)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/whisper/__init__.py", line 155, in load_model
    model.load_state_dict(checkpoint["model_state_dict"])
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 2584, in load_state_dict
    raise RuntimeError(
RuntimeError: Error(s) in loading state_dict for Whisper:
	While copying the parameter named "encoder.blocks.0.attn.query.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.0.attn.key.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.0.attn.value.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.0.attn.out.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.0.mlp.0.weight", whose dimensions in the model are torch.Size([5120, 1280]) and whose dimensions in the checkpoint are torch.Size([5120, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.0.mlp.2.weight", whose dimensions in the model are torch.Size([1280, 5120]) and whose dimensions in the checkpoint are torch.Size([1280, 5120]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.1.attn.query.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.1.attn.key.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.1.attn.value.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.1.attn.out.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.1.mlp.0.weight", whose dimensions in the model are torch.Size([5120, 1280]) and whose dimensions in the checkpoint are torch.Size([5120, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.1.mlp.2.weight", whose dimensions in the model are torch.Size([1280, 5120]) and whose dimensions in the checkpoint are torch.Size([1280, 5120]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.2.attn.query.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.2.attn.key.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.2.attn.value.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.2.attn.out.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.2.mlp.0.weight", whose dimensions in the model are torch.Size([5120, 1280]) and whose dimensions in the checkpoint are torch.Size([5120, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.2.mlp.2.weight", whose dimensions in the model are torch.Size([1280, 5120]) and whose dimensions in the checkpoint are torch.Size([1280, 5120]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.3.attn.query.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.3.attn.key.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.3.attn.value.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.3.attn.out.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.3.mlp.0.weight", whose dimensions in the model are torch.Size([5120, 1280]) and whose dimensions in the checkpoint are torch.Size([5120, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.3.mlp.2.weight", whose dimensions in the model are torch.Size([1280, 5120]) and whose dimensions in the checkpoint are torch.Size([1280, 5120]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.4.attn.query.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.4.attn.key.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.4.attn.value.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.4.attn.out.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.4.mlp.0.weight", whose dimensions in the model are torch.Size([5120, 1280]) and whose dimensions in the checkpoint are torch.Size([5120, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.4.mlp.2.weight", whose dimensions in the model are torch.Size([1280, 5120]) and whose dimensions in the checkpoint are torch.Size([1280, 5120]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.5.attn.query.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.5.attn.key.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.5.attn.value.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.5.attn.out.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.5.mlp.0.weight", whose dimensions in the model are torch.Size([5120, 1280]) and whose dimensions in the checkpoint are torch.Size([5120, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.5.mlp.2.weight", whose dimensions in the model are torch.Size([1280, 5120]) and whose dimensions in the checkpoint are torch.Size([1280, 5120]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.6.attn.query.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.6.attn.key.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.6.attn.value.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.6.attn.out.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.6.mlp.0.weight", whose dimensions in the model are torch.Size([5120, 1280]) and whose dimensions in the checkpoint are torch.Size([5120, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.6.mlp.2.weight", whose dimensions in the model are torch.Size([1280, 5120]) and whose dimensions in the checkpoint are torch.Size([1280, 5120]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.7.attn.query.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.7.attn.key.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.7.attn.value.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.7.attn.out.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.7.mlp.0.weight", whose dimensions in the model are torch.Size([5120, 1280]) and whose dimensions in the checkpoint are torch.Size([5120, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.7.mlp.2.weight", whose dimensions in the model are torch.Size([1280, 5120]) and whose dimensions in the checkpoint are torch.Size([1280, 5120]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.8.attn.query.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.8.attn.key.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.8.attn.value.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.8.attn.out.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.8.mlp.0.weight", whose dimensions in the model are torch.Size([5120, 1280]) and whose dimensions in the checkpoint are torch.Size([5120, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.8.mlp.2.weight", whose dimensions in the model are torch.Size([1280, 5120]) and whose dimensions in the checkpoint are torch.Size([1280, 5120]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.9.attn.query.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.9.attn.key.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.9.attn.value.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.9.attn.out.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.9.mlp.0.weight", whose dimensions in the model are torch.Size([5120, 1280]) and whose dimensions in the checkpoint are torch.Size([5120, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.9.mlp.2.weight", whose dimensions in the model are torch.Size([1280, 5120]) and whose dimensions in the checkpoint are torch.Size([1280, 5120]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.10.attn.query.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.10.attn.key.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.10.attn.value.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.10.attn.out.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.10.mlp.0.weight", whose dimensions in the model are torch.Size([5120, 1280]) and whose dimensions in the checkpoint are torch.Size([5120, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.10.mlp.2.weight", whose dimensions in the model are torch.Size([1280, 5120]) and whose dimensions in the checkpoint are torch.Size([1280, 5120]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.11.attn.query.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.11.attn.key.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.11.attn.value.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.11.attn.out.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.11.mlp.0.weight", whose dimensions in the model are torch.Size([5120, 1280]) and whose dimensions in the checkpoint are torch.Size([5120, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.11.mlp.2.weight", whose dimensions in the model are torch.Size([1280, 5120]) and whose dimensions in the checkpoint are torch.Size([1280, 5120]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.12.attn.query.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.12.attn.key.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.12.attn.value.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.12.attn.out.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.12.mlp.0.weight", whose dimensions in the model are torch.Size([5120, 1280]) and whose dimensions in the checkpoint are torch.Size([5120, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.12.mlp.2.weight", whose dimensions in the model are torch.Size([1280, 5120]) and whose dimensions in the checkpoint are torch.Size([1280, 5120]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.13.attn.query.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.13.attn.key.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.13.attn.value.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.13.attn.out.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.13.mlp.0.weight", whose dimensions in the model are torch.Size([5120, 1280]) and whose dimensions in the checkpoint are torch.Size([5120, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.13.mlp.2.weight", whose dimensions in the model are torch.Size([1280, 5120]) and whose dimensions in the checkpoint are torch.Size([1280, 5120]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.14.attn.query.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.14.attn.key.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.14.attn.value.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.14.attn.out.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.14.mlp.0.weight", whose dimensions in the model are torch.Size([5120, 1280]) and whose dimensions in the checkpoint are torch.Size([5120, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.14.mlp.2.weight", whose dimensions in the model are torch.Size([1280, 5120]) and whose dimensions in the checkpoint are torch.Size([1280, 5120]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.15.attn.query.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.15.attn.key.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.15.attn.value.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.15.attn.out.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.15.mlp.0.weight", whose dimensions in the model are torch.Size([5120, 1280]) and whose dimensions in the checkpoint are torch.Size([5120, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.15.mlp.2.weight", whose dimensions in the model are torch.Size([1280, 5120]) and whose dimensions in the checkpoint are torch.Size([1280, 5120]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.16.attn.query.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.16.attn.key.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.16.attn.value.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.16.attn.out.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.16.mlp.0.weight", whose dimensions in the model are torch.Size([5120, 1280]) and whose dimensions in the checkpoint are torch.Size([5120, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.16.mlp.2.weight", whose dimensions in the model are torch.Size([1280, 5120]) and whose dimensions in the checkpoint are torch.Size([1280, 5120]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.17.attn.query.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.17.attn.key.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.17.attn.value.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.17.attn.out.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.17.mlp.0.weight", whose dimensions in the model are torch.Size([5120, 1280]) and whose dimensions in the checkpoint are torch.Size([5120, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.17.mlp.2.weight", whose dimensions in the model are torch.Size([1280, 5120]) and whose dimensions in the checkpoint are torch.Size([1280, 5120]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.18.attn.query.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.18.attn.key.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.18.attn.value.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.18.attn.out.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.18.mlp.0.weight", whose dimensions in the model are torch.Size([5120, 1280]) and whose dimensions in the checkpoint are torch.Size([5120, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.18.mlp.2.weight", whose dimensions in the model are torch.Size([1280, 5120]) and whose dimensions in the checkpoint are torch.Size([1280, 5120]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.19.attn.query.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.19.attn.key.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.19.attn.value.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.19.attn.out.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.19.mlp.0.weight", whose dimensions in the model are torch.Size([5120, 1280]) and whose dimensions in the checkpoint are torch.Size([5120, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.19.mlp.2.weight", whose dimensions in the model are torch.Size([1280, 5120]) and whose dimensions in the checkpoint are torch.Size([1280, 5120]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.20.attn.query.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.20.attn.key.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.20.attn.value.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.20.attn.out.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.20.mlp.0.weight", whose dimensions in the model are torch.Size([5120, 1280]) and whose dimensions in the checkpoint are torch.Size([5120, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.20.mlp.2.weight", whose dimensions in the model are torch.Size([1280, 5120]) and whose dimensions in the checkpoint are torch.Size([1280, 5120]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.21.attn.query.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.21.attn.key.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.21.attn.value.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.21.attn.out.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.21.mlp.0.weight", whose dimensions in the model are torch.Size([5120, 1280]) and whose dimensions in the checkpoint are torch.Size([5120, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.21.mlp.2.weight", whose dimensions in the model are torch.Size([1280, 5120]) and whose dimensions in the checkpoint are torch.Size([1280, 5120]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.22.attn.query.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.22.attn.key.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.22.attn.value.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.22.attn.out.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.22.mlp.0.weight", whose dimensions in the model are torch.Size([5120, 1280]) and whose dimensions in the checkpoint are torch.Size([5120, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.22.mlp.2.weight", whose dimensions in the model are torch.Size([1280, 5120]) and whose dimensions in the checkpoint are torch.Size([1280, 5120]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.23.attn.query.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.23.attn.key.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.23.attn.value.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.23.attn.out.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.23.mlp.0.weight", whose dimensions in the model are torch.Size([5120, 1280]) and whose dimensions in the checkpoint are torch.Size([5120, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.23.mlp.2.weight", whose dimensions in the model are torch.Size([1280, 5120]) and whose dimensions in the checkpoint are torch.Size([1280, 5120]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.24.attn.query.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.24.attn.key.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.24.attn.value.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.24.attn.out.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.24.mlp.0.weight", whose dimensions in the model are torch.Size([5120, 1280]) and whose dimensions in the checkpoint are torch.Size([5120, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.24.mlp.2.weight", whose dimensions in the model are torch.Size([1280, 5120]) and whose dimensions in the checkpoint are torch.Size([1280, 5120]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.25.attn.query.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.25.attn.key.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.25.attn.value.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.25.attn.out.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.25.mlp.0.weight", whose dimensions in the model are torch.Size([5120, 1280]) and whose dimensions in the checkpoint are torch.Size([5120, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.25.mlp.2.weight", whose dimensions in the model are torch.Size([1280, 5120]) and whose dimensions in the checkpoint are torch.Size([1280, 5120]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.26.attn.query.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.26.attn.key.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.26.attn.value.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.26.attn.out.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.26.mlp.0.weight", whose dimensions in the model are torch.Size([5120, 1280]) and whose dimensions in the checkpoint are torch.Size([5120, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.26.mlp.2.weight", whose dimensions in the model are torch.Size([1280, 5120]) and whose dimensions in the checkpoint are torch.Size([1280, 5120]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.27.attn.query.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.27.attn.key.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.27.attn.value.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.27.attn.out.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.27.mlp.0.weight", whose dimensions in the model are torch.Size([5120, 1280]) and whose dimensions in the checkpoint are torch.Size([5120, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.27.mlp.2.weight", whose dimensions in the model are torch.Size([1280, 5120]) and whose dimensions in the checkpoint are torch.Size([1280, 5120]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.28.attn.query.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.28.attn.key.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.28.attn.value.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.28.attn.out.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.28.mlp.0.weight", whose dimensions in the model are torch.Size([5120, 1280]) and whose dimensions in the checkpoint are torch.Size([5120, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.28.mlp.2.weight", whose dimensions in the model are torch.Size([1280, 5120]) and whose dimensions in the checkpoint are torch.Size([1280, 5120]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.29.attn.query.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.29.attn.key.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.29.attn.value.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.29.attn.out.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.29.mlp.0.weight", whose dimensions in the model are torch.Size([5120, 1280]) and whose dimensions in the checkpoint are torch.Size([5120, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.29.mlp.2.weight", whose dimensions in the model are torch.Size([1280, 5120]) and whose dimensions in the checkpoint are torch.Size([1280, 5120]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.30.attn.query.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.30.attn.key.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.30.attn.value.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.30.attn.out.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.30.mlp.0.weight", whose dimensions in the model are torch.Size([5120, 1280]) and whose dimensions in the checkpoint are torch.Size([5120, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.30.mlp.2.weight", whose dimensions in the model are torch.Size([1280, 5120]) and whose dimensions in the checkpoint are torch.Size([1280, 5120]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.31.attn.query.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.31.attn.key.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.31.attn.value.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.31.attn.out.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.31.mlp.0.weight", whose dimensions in the model are torch.Size([5120, 1280]) and whose dimensions in the checkpoint are torch.Size([5120, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "encoder.blocks.31.mlp.2.weight", whose dimensions in the model are torch.Size([1280, 5120]) and whose dimensions in the checkpoint are torch.Size([1280, 5120]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "decoder.blocks.0.attn.query.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "decoder.blocks.0.attn.key.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "decoder.blocks.0.attn.value.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "decoder.blocks.0.attn.out.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "decoder.blocks.0.cross_attn.query.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "decoder.blocks.0.cross_attn.key.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "decoder.blocks.0.cross_attn.value.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "decoder.blocks.0.cross_attn.out.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "decoder.blocks.0.mlp.0.weight", whose dimensions in the model are torch.Size([5120, 1280]) and whose dimensions in the checkpoint are torch.Size([5120, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "decoder.blocks.0.mlp.2.weight", whose dimensions in the model are torch.Size([1280, 5120]) and whose dimensions in the checkpoint are torch.Size([1280, 5120]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "decoder.blocks.1.attn.query.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "decoder.blocks.1.attn.key.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "decoder.blocks.1.attn.value.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "decoder.blocks.1.attn.out.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "decoder.blocks.1.cross_attn.query.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "decoder.blocks.1.cross_attn.key.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "decoder.blocks.1.cross_attn.value.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "decoder.blocks.1.cross_attn.out.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "decoder.blocks.1.mlp.0.weight", whose dimensions in the model are torch.Size([5120, 1280]) and whose dimensions in the checkpoint are torch.Size([5120, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "decoder.blocks.1.mlp.2.weight", whose dimensions in the model are torch.Size([1280, 5120]) and whose dimensions in the checkpoint are torch.Size([1280, 5120]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "decoder.blocks.2.attn.query.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "decoder.blocks.2.attn.key.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "decoder.blocks.2.attn.value.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "decoder.blocks.2.attn.out.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "decoder.blocks.2.cross_attn.query.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "decoder.blocks.2.cross_attn.key.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "decoder.blocks.2.cross_attn.value.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "decoder.blocks.2.cross_attn.out.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "decoder.blocks.2.mlp.0.weight", whose dimensions in the model are torch.Size([5120, 1280]) and whose dimensions in the checkpoint are torch.Size([5120, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "decoder.blocks.2.mlp.2.weight", whose dimensions in the model are torch.Size([1280, 5120]) and whose dimensions in the checkpoint are torch.Size([1280, 5120]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "decoder.blocks.3.attn.query.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "decoder.blocks.3.attn.key.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "decoder.blocks.3.attn.value.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "decoder.blocks.3.attn.out.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "decoder.blocks.3.cross_attn.query.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "decoder.blocks.3.cross_attn.key.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "decoder.blocks.3.cross_attn.value.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "decoder.blocks.3.cross_attn.out.weight", whose dimensions in the model are torch.Size([1280, 1280]) and whose dimensions in the checkpoint are torch.Size([1280, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "decoder.blocks.3.mlp.0.weight", whose dimensions in the model are torch.Size([5120, 1280]) and whose dimensions in the checkpoint are torch.Size([5120, 1280]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).
	While copying the parameter named "decoder.blocks.3.mlp.2.weight", whose dimensions in the model are torch.Size([1280, 5120]) and whose dimensions in the checkpoint are torch.Size([1280, 5120]), an exception occurred : ('HIP error: invalid device function\nHIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing AMD_SERIALIZE_KERNEL=3\nCompile with `TORCH_USE_HIP_DSA` to enable device-side assertions.\n',).

[glangford]

My graphics card is Radeon RX 7600

The RX 7600 is not officially supported according to this AMD ROCm 6.2.4 system requirements page.

https://rocm.docs.amd.com/projects/install-on-linux/en/latest/reference/system-requirements.html

[viktorprogger]

Finally I solved this with env variable HSA_OVERRIDE_GFX_VERSION set to 11.0.0. Whisper works on GPU, and I have no more problems.
I still don't know why 11.0.2 doesn't work, but I'm glad I can use GPU for transcription.

P. S. This solution helps me to start ollama models on GPU too.

[alanliongar]

It seems like you have to put the env variable HAS_OVERRIDE_GFX_VERSION set to a specific number, based upon your videoboard. In my case the env setted up to 10.3.0 works with the RX 6600. I spent the whole day here trying to figure out what was happening before finding the @viktorprogger solve, thank you so much my friend.