Failure to Initialize Container: unsatisfied condition: cuda>=12.6

[Jack-Khuu]

GPU based workflows fail after bumping Cuda requirements from 12.4 -> 12.6 in torchchat.
Would love some help updating the driver or suggestions on how to update test config

Example Run: https://github.com/pytorch/torchchat/actions/runs/14053926197/job/39349546495

docker: Error response from daemon: failed to create task for container: failed to create shim task: 
OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running prestart hook #0: exit status 1, stdout: , 

stderr: Auto-detected mode as 'legacy'
nvidia-container-cli: requirement error: unsatisfied condition: cuda>=12.6, 
please update your driver to a newer version, or use an earlier cuda container: unknown.

Test Runner Config: https://github.com/pytorch/torchchat/blob/fea361f6cce0b1cdd54cc211dde19266753b60fc/.github/workflows/more-tests.yml#L11-L19

  test-cuda:
    permissions:
      id-token: write
      contents: read
    uses: pytorch/test-infra/.github/workflows/linux_job_v2.yml@main
    with:
      runner: linux.g5.4xlarge.nvidia.gpu
      gpu-arch-type: cuda
      gpu-arch-version: "12.6"

Similar related past Issue: #5191

[clee2000]

cc @atalman ? @seemethere I'm not sure who owns nova

I think it needs to use a newer cuda driver which would require changes to linux_job_v2 to take in an input for this, but I also see pytorch/pytorch jobs named cuda12.6 but they show 12.4 when nvidia-smi is run. Do the 12.6 on pytorch/pytorch just mean that the binary was built with 12.6 but not necessarily run on 12.6?

[HonestDeng]

cc @atalman ? @seemethere I'm not sure who owns nova

I think it needs to use a newer cuda driver which would require changes to linux_job_v2 to take in an input for this, but I also see pytorch/pytorch jobs named cuda12.6 but they show 12.4 when nvidia-smi is run. Do the 12.6 on pytorch/pytorch just mean that the binary was built with 12.6 but not necessarily run on 12.6?

Hi. I'm the owner of Update CI Jobs in anticipation for Cuda 12.4 deprecation pytorch/torchchat#1515.

I'm new to torchchat and confused what you said. Do you mean we should update linux_job_v2.yml to use a newer cuda driver?

Thanks.

[Jack-Khuu]

Ah, thanks @clee2000

@HonestDeng I'll follow up with you on Discord 😃