vpn site2site is failing in travis builds

[powellquiring]

Test 8 has been failing most builds, Example

https://app.travis-ci.com/github/IBM-Cloud/vpc-tutorials/builds/253652512

SCENARIO=vpc-site2site-vpn TEST=tests/vpc-site2site-vpn/create-with-terraform.sh

=================================== FAILURES ===================================
___________________________________ test_dns ___________________________________
    def test_dns():
      for connection in (connection_ip_fip_bastion_to_ip_private_cloud, connection_ip_fip_onprem):
        with connection() as c:
          ret = c.run(f"/usr/bin/dig +short {global_cache.hostname_postgresql}", in_stream=False)
>         assert global_cache.ip_endpoint_gateway_postgresql == ret.stdout.strip()
E         AssertionError: assert '10.1.1.10' == 'icd-prod-us-...\n166.9.12.32'
E           + 10.1.1.10
E           - icd-prod-us-south-db-751334.us-south.serviceendpoint.cloud.ibm.com.
E           - 166.9.14.21
E           - 166.9.16.26
E           - 166.9.12.32
test/test_scenario.py:100: AssertionError
=============================== warnings summary ===============================
../../../../usr/lib/python3.9/site-packages/invoke/loader.py:3
  /usr/lib/python3.9/site-packages/invoke/loader.py:3: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
    import imp
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
FAILED test/test_scenario.py::test_dns - AssertionError: assert '10.1.1.10' =...
==================== 1 failed, 4 passed, 1 warning in 9.80s ====================

[powellquiring]

Not sure, added a sleep before running terraform a second time to allow the endpoint gateway to get a value. Comment in the TEST=tests/vpc-site2site-vpn/create-with-terraform.sh

# Testing, test_dns, is verifying the DNS resolution on-prem matches the endpoint gateways and this is failing
# for two different reasons.
# 1. Problem: terraform output ip_endpoint_gateway_cos was 0.0.0.0
#    ibm_is_virtual_endpoint_gateway.cos.ips[0].address
#    Fix: extra time, run terraform apply again, and the endpoint will have an address
# 2. dig from either the cloud or onprem is returning the public addresses: 166.* 
#    def test_dns():
#      for connection in (connection_ip_fip_bastion_to_ip_private_cloud, connection_ip_fip_onprem):
#        with connection() as c:
#          ret = c.run(f"/usr/bin/dig +short {global_cache.hostname_postgresql}", in_stream=False)
#>         assert global_cache.ip_endpoint_gateway_postgresql == ret.stdout.strip()
#E         AssertionError: assert '10.1.1.9' == 'icd-prod-us-...7\n166.9.90.6'
#E           + 10.1.1.9
#E           - icd-prod-us-south-db-711595.us-south.serviceendpoint.cloud.ibm.com.
#E           - 166.9.48.66
#E           - 166.9.58.127
#E           - 166.9.90.6
#      Fix: I do not know
sleep 60
terraform apply

problem 2 did not reproduce on my desktop

[powellquiring]

If this fails again, unroll the loop into two test functions to pinpoint the source of the problem