Improve Longevity Test

[pleshakov]

As a maintainer of NGF
I want our longevity test to more closely mimic a real customer environment
So that we can discover bugs before they reach our users.

Acceptance Criteria

• A second replica is created to test leader election.
• The cluster name and NGF pod name are present to the generated output.

[bjee19]

With the changes introduced in our control plane data plane split and the updates to the test in #3305, all of the nginx metrics tracked in the GKE dashboard are unavailable. We think that this is because we stopped using the NGINX Prometheus Exporter, which disqualifies us from accessing the metrics shown here https://cloud.google.com/monitoring/api/metrics_kubernetes#kubernetes-nginx.

For us to access the metrics from agent, we would need to set up some GKE specific service which would cost money.

An alternative, and possible addition to the acceptance criteria, is to set up OSS prometheus in the cluster while running the longevity tests, scrape that, and create similar graphs. We may be able to recover some of the lost nginx metrics.

[sjberman]

Some additional changes I think we need (more-so reliability improvements) would be to run over 3 days instead of 4, and run with 5 nodes instead of the default of 3.