chore(ci_visibility): retry Test Optimization API calls on server errors

[vitor-de-araujo]

Description

Currently, if a request to the Test Optimization settings API fails with a 500 or other server error, we do not try again, and run the test session with Test Optimization features disabled. This PR adds retries to the Test Optimization API requests, for server errors (status >= 500; it does not make sense to retry 4xx errors as they are unlikely to succeed if called again with the same parameters) and network errors (such as timeout).

Testing

Unit tests added.

Risks

None.

Additional Notes

None.

[github-actions]

CODEOWNERS have been resolved as:

ddtrace/internal/ci_visibility/_api_client.py                           @DataDog/ci-app-libraries
ddtrace/internal/ci_visibility/telemetry/api_request.py                 @DataDog/ci-app-libraries
tests/ci_visibility/api_client/test_ci_visibility_api_client.py         @DataDog/ci-app-libraries

[github-actions]

Bootstrap import analysis

Comparison of import times between this PR and base.

Summary

The average import time from this PR is: 221 ± 4 ms.

The average import time from base is: 221 ± 2 ms.

The import time difference between this PR and base is: 0.1 ± 0.1 ms.

The difference is not statistically significant (z = 0.84).

Import time breakdown

The following import paths have shrunk:

ddtrace.auto 1.445 ms (0.65%)

ddtrace.bootstrap.sitecustomize 0.936 ms (0.42%)

ddtrace.bootstrap.preload 0.936 ms (0.42%)

ddtrace.internal.remoteconfig.client 0.487 ms (0.22%)

ddtrace 0.509 ms (0.23%)

ddtrace.internal._unpatched 0.037 ms (0.02%)

json 0.037 ms (0.02%)

json.decoder 0.037 ms (0.02%)

re 0.037 ms (0.02%)

enum 0.037 ms (0.02%)

types 0.037 ms (0.02%)

[gnufede]

nit: since the code allows for >500, why not add 504 or other status codes here?