Fix Teardown and Collection of Results of Longevity Tests

[mpstefan]

Currently, during the teardown and collection of our longevity tests, sometimes, no results are recorded for the run. We need an investigation into why this is occurring and a fix before 2.1s release.

Acceptance

• Consistent recording of longevity test results.
• General formatting of results and naming of files should be corrected

Possible Causes

• Kubernetes connection errors
• Failure to collect traffic summary
• Failure to collect logs

[sjberman]

Tests seem to be running properly now. Issues from before were likely due to cancelling the tests early.

[bjee19]

Do you think its worth adding any guardrails against the errors on early cancellation?

[sjberman]

The wrk process won't report until it's done, so if someone kills the test in the middle, there won't be any results to gather. That's really the issues we ran into last time. Now I'm not sure what caused the issues in getting logs from GCP, could've been networking issues, but I couldn't get logs even locally for multiple hours, so it wasn't much of a retryable event unfortunately. I think we keep an eye on this in the future. If it continues to be a problem, we can look into better solutions.

[bjee19]

Ok sure that sounds good, thanks for looking into it.