Update benchmark_step.jl for CUDA benchmarking with useful kernel names

[petebachant]

This uses CliMA/ClimaCore.jl#2376 to provide more useful CUDA kernel names in benchmarks.

TODO

• Can we format the kernel names with anything other than underscores as separators? --> No, all non-alphanumeric characters get converted to underscores.
• Do we want to do this on the benchmarks that use benchmark.jl, not benchmark_step.jl?
• Update Buildkite pipeline to use this feature
• Switch to non-dev ClimaCore

[petebachant]

These changes were made by a YAML auto-formatter in VS Code. Is there a style guide I might be breaking here?

[imreddyTeja]

I'm not sure... this is something I have been wondering as well. I considered following this example, which is used in Buildkite's docs.

[petebachant]

@dennisYatunin @imreddyTeja thoughts on the timeouts here? Should I increase the limit or disable kernel renaming?

[imreddyTeja]

@dennisYatunin @imreddyTeja thoughts on the timeouts here? Should I increase the limit or disable kernel renaming?

What is the advantage of using kernel renaming in climaatmos-ci for buildkite steps that don't profile? Is the idea to run the profiler at the end of each simulation?

Comparing the buildkite for this PR to the main's last buildkite run shows a ~40% slowdown, but everything after the first step doesn't seem to be affected. I'm not sure if that cost is worth it at the moment.

[petebachant]

What is the advantage of using kernel renaming in climaatmos-ci for buildkite steps that don't profile? Is the idea to run the profiler at the end of each simulation?

I believe that was Dennis's vision, and then we'd process and summarize all of the profiling results together in a later step.