Fix Evoformer compilation

[sdvillal]

EvoformerAttnBuilder has some problems which preclude compiling the extension on several scenarios (e.g., isolated conda environment with cuda toolchain, lack of hardware in the system) and breaks some standard DeepSpeed configuration of target capabilities.

Changes

• Fix evoformer CUTLASS detection:

  • Allow to skip it, useful when CUTLASS is already correctly setup (e.g., in a conda environment with CUTLASS and the CUDA toolchain)
  • Fix misleading use of deprecated nvidia-cutlass pypi package by actually using the provided bindings but discouraging this route as these bindings are not maintained anymore

• Fix evoformer compilation with no GPU is present:

  • this is taken care correctly and more generally by builder.compute_capability_args
  • allow for cross-compilation in systems without GPU
  • allows for compilation against all available virtual architectures and binary outputs
  • see e.g., [REQUEST] build prebuilt wheels #5308

• Make all these changes configurable and explicit through documented environment variables

Tested in all scenarios.

[sdvillal]

Tested and ready for review @loadams @tjruwase

[tohtana]

Hi @sdvillal
Thank you for your contribution! This looks good to me. Can you just fix the format? It seems I don't have write access to your branch.

By the way, test_DS4Sci_EvoformerAttention shows some mismatch. This is unrelated to your change, but have you experieced it?

Test	Status
test_DS4Sci_EvoformerAttention[tensor_shape0-dtype1] (bfloat16, 256×256)	PASSED
test_DS4Sci_EvoformerAttention[tensor_shape1-dtype0] (float16, 512×256)	PASSED
test_DS4Sci_EvoformerAttention[tensor_shape1-dtype1] (bfloat16, 512×256)	FAILED

[sdvillal]

Thanks a lot for the quick review and merge @tohtana!

I have fixed formatting (sorry about it, one should read the contributing guidelines before contributing...).

I have not personally experienced the mismatch. We have been running on:

• A100 GPUs, for which the extension was developed for (a quick peek online [1, 2] seems to indicate this is a Hopper problem?)
• only very recently we are also using B300s, where we have not yet experienced troubles
• only using modern/latest dependencies (particularly, I feel maybe relevant, cutlass)

I could try to run the test a few times in this context and see if it happens for me, could that info be useful?

In any case, I feel the extension is showing its age and it might require some love to these GEMMs and generally to make it worthwhile to use on Hopper and newer.

[tohtana]

@sdvillal Yes, I encountered the issue with H100. The test doesn't throw an error with L40S on our CI.
At the moment we don't really have the bandwidth/expertise on our side to chase down and fix an H100/Hopper-specific issue, so I'd suggest we leave it as-is for now.

[Flamefire]

After this it doesn't work at all for me anymore:

One issue without this PR is with e.g. test_DS4Sci_EvoformerAttention[tensor_shape0-dtype0] on A100: Depending on the TORCH_CUDA_ARCHLIST used at build time (with DS_BUILD_OPS) it fails with a cudaErrorInvalidDeviceFunction:

• TORCH_CUDA_ARCHLIST=7.0;8.0 works
• TORCH_CUDA_ARCHLIST=8.0;7.0 fails

We want to use the same installation also for older GPUs and this failure is very unexpected as the order shouldn't matter.

With this PR it doesn't compile anymore because the include_paths function is supposed to return a list of strings not Path instances. self.strip_empty_entries then fails calling len on it.

How did that work for you? I don't see how it could ever succeed.

Proposed fix for that in #7862

[sdvillal]

Thanks for the report and the fix @Flamefire, and apologies that this broke compilation for manually defined paths - I believe testing this case likely slipped my attention as I was getting the PR ready for review and it is not being tested by the CIs? I will give a hand with the PR, I suggest we propose a test to avoid this happening again.

Regarding the order issue, do you think it is also related to the changes in this PR?

[Flamefire]

Thanks for the report and the fix @Flamefire, and apologies that this broke compilation for manually defined paths - I believe testing this case likely slipped my attention as I was getting the PR ready for review and it is not being tested by the CIs? I will give a hand with the PR, I suggest we propose a test to avoid this happening again.

The test seems to not be run on CI and nothing on CI seems to build this kernel/op at all.
There is a CI workflow that can manually be triggered that would run this test but it has been excluded recently

Regarding the order issue, do you think it is also related to the changes in this PR?

No, I noticed the issue since at least 0.14.5, way before this PR. But it persists on current master

[sdvillal]

As a workaround, would pointing to the relevant include dirs (e.g., using CPATH) and removing deepspeed specific configuration (i.e., setting CUTLASS_PATH=DS_IGNORE_CUTLASS_DETECTION) work for you?

[Flamefire]

Workaround for what exactly? This Path issue?
Then yes I guess setting CPATH and then CUTLASS_PATH=DS_IGNORE_CUTLASS_DETECTION could avoid triggering returning of Path instances

Doesn't help for the arch issues of course

[sdvillal]

I would recommend opening a separate issue for the ordering problem, if there is no one already, as it is unrelated to the changes in this PR.

[Flamefire]

Done: #7863