Improve OSU benchmark to better detect system jitters

[hanwen-cluster]

Description of changes

This PR includes improvements and fixes to the OSU benchmark integration tests:

• Remove devsettings to disable in_place_update_on_fleet in OSU test

• Retry OSU benchmark once in case of failure

• Run OSU alltoall only on small clusters

• Improve performance test README

• Skip OSU pt2pt benchmarks on large clusters

• Add barrier benchmark to OSU test

• Run OSU on large clusters with 500 compute nodes

• Only compare with baseline if baseline file exists

• Remove OSU result upload to CloudWatch metrics

See commits descriptions for more details.

Tests

The following test has passed:

test-suites:
  performance_tests:
    test_osu.py::test_osu:
      dimensions:
        - regions: [ {{ c5_xlarge_CAPACITY_RESERVATION_510_INSTANCES_2_HOURS_YESPG_alinux2023 }} ]
          instances: [ "c5.xlarge" ]
          oss: ["alinux2023"] # ParallelCluster does not release official Rocky images. Skip the test.
          schedulers: [ "slurm" ]

References

This change is able to reproduce these issues:
#7095
#6449

Checklist

• Make sure you are pointing to the right branch.
• If you're creating a patch for a branch other than develop add the branch name as prefix in the PR title (e.g. [release-3.6]).
• Check all commits' messages are clear, describing what and why vs how.
• Make sure to have added unit tests or integration tests to cover the new/modified code.
• Check if documentation is impacted by this change.

Please review the guidelines for contributing and Pull Request Instructions.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.