ci: handle RemoteDisconnected in snapshot tests (#9464)

This change handles errors [like
this](https://app.circleci.com/pipelines/github/DataDog/dd-trace-py/62764/workflows/72b00236-192f-488e-8364-da5f22bb4427/jobs/3909294)
in the snapshot test harness by retrying requests. This makes sense
because we always expect such errors to be transient, and if they happen
not to be the test will eventually time out.

This change also marks a few recently observed unreliable failures
([one](https://app.circleci.com/pipelines/github/DataDog/dd-trace-py/62774/workflows/d359abbe-c28d-4618-87a0-da264669268a/jobs/3910041),
[two](https://app.circleci.com/pipelines/github/DataDog/dd-trace-py/62597/workflows/f12d4201-ebf6-4037-bc53-7fbe35c4a8c9/jobs/3901172))
and adds an exception condition to a Django test that [recently
exhibited
it](https://app.circleci.com/pipelines/github/DataDog/dd-trace-py/62723/workflows/56187749-f64c-4fb9-918a-9d899f8c611f/jobs/3907658).

## Checklist

- [ ] Change(s) are motivated and described in the PR description
- [ ] Testing strategy is described if automated tests are not included
in the PR
- [ ] Risks are described (performance impact, potential for breakage,
maintainability)
- [ ] Change is maintainable (easy to change, telemetry, documentation)
- [ ] [Library release note
guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html)
are followed or label `changelog/no-changelog` is set
- [ ] Documentation is included (in-code, generated user docs, [public
corp docs](https://github.com/DataDog/documentation/))
- [ ] Backport labels are set (if
[applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting))
- [ ] If this PR changes the public interface, I've notified
`@DataDog/apm-tees`.

## Reviewer Checklist

- [x] Title is accurate
- [x] All changes are related to the pull request's stated goal
- [x] Description motivates each change
- [x] Avoids breaking
[API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces)
changes
- [x] Testing strategy adequately addresses listed risks
- [x] Change is maintainable (easy to change, telemetry, documentation)
- [x] Release note makes sense to a user of the library
- [x] Author has acknowledged and discussed the performance implications
of this PR as reported in the benchmarks PR comment
- [x] Backport labels are set in a manner that is consistent with the
[release branch maintenance
policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)

Author: emmettbutler

tests/contrib/django_celery/test_django_celery.py

@@ -34,4 +34,6 @@ def test_django_celery_gevent_startup():
         assert "celery@" in out, "Celery started correctly"
         assert "DJANGO_SETTINGS_MODULE" not in err, "No Django lazy objects"
     else:
-        assert retcode == 0, "Celery was finished with errors: %s" % err.decode("utf-8")
+        err_text = err.decode("utf-8")
+        if "not recommended" not in err_text:
+            assert retcode == 0, "Celery was finished with errors: %s" % err_text

tests/debugging/test_debugger.py

@@ -618,6 +618,7 @@ def test_debugger_wrapped_function_on_function_probe(stuff):
     assert g is not f
 
 
+@flaky(1735812000)
 def test_debugger_line_probe_on_wrapped_function(stuff):
     wrapt.wrap_function_wrapper(stuff, "Stuff.instancestuff", wrapper)
 

tests/utils.py

@@ -1018,8 +1018,16 @@ def snapshot_context(
         except Exception as e:
             pytest.fail("Could not connect to test agent: %s" % str(e), pytrace=False)
         else:
-            r = conn.getresponse()
-            if r.status != 200:
+            r = None
+            attempt_start = time.time()
+            while r is None and time.time() - attempt_start < 60:
+                try:
+                    r = conn.getresponse()
+                except http.client.RemoteDisconnected:
+                    time.sleep(1)
+            if r is None:
+                pytest.fail("Repeated attempts to start testagent session failed", pytrace=False)
+            elif r.status != 200:
                 # The test agent returns nice error messages we can forward to the user.
                 pytest.fail(to_unicode(r.read()), pytrace=False)