Improve robustness of get_pod_ip() function

[bshephar]

In cases where single-stack IPv6 is used, it's possible that an unhandled exception is raised since we first attempt to verify ipv4 connectivity and fail before checking ipv6. This change makes the get_pod_ip() function more robust by ensuring that both are checked before raising an error.

Fixes #478

[openshift-ci]

Hi @bshephar. Thanks for your PR.

I'm waiting for a openstack-k8s-operators member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[bshephar]

Tested like so:

>>> def get_pod_ip():
...     import socket
...
...     ipstacks = [socket.AF_INET, socket.AF_INET6]
...     last_error = None
...
...     host = "google.com"
...     port = 443
...
...     for ipstack in ipstacks:
...         try:
...             s = socket.socket(ipstack, socket.SOCK_DGRAM)
...             s.settimeout(2)
...             s.connect((host, port))
...             ip = s.getsockname()[0]
...             s.close()
...             return ip if ipstack == socket.AF_INET else f"[{ip}]"
...         except Exception as e:
...             last_error = e
...
...     raise RuntimeError(f"Could not determine pod IP: {last_error}")
...
>>> get_pod_ip()
'[2403:5817:3864:0:e0bd:373d:1b4e:1d5c]'

[bshephar]

My bad:

++ /usr/bin/python3 /usr/bin/manage.py collectstatic --noinput --clear
Traceback (most recent call last):
  File "/usr/bin/manage.py", line 23, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/lib/python3.9/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/usr/lib/python3.9/site-packages/django/core/management/__init__.py", line 363, in execute
    settings.INSTALLED_APPS
  File "/usr/lib/python3.9/site-packages/django/conf/__init__.py", line 82, in __getattr__
    self._setup(name)
  File "/usr/lib/python3.9/site-packages/django/conf/__init__.py", line 69, in _setup
    self._wrapped = Settings(settings_module)
  File "/usr/lib/python3.9/site-packages/django/conf/__init__.py", line 170, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/lib64/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/usr/lib/python3.9/site-packages/openstack_dashboard/settings.py", line 245, in <module>
    from local.local_settings import *  # noqa: F403,H303
  File "/usr/lib/python3.9/site-packages/openstack_dashboard/local/local_settings.py", line 86, in <module>
    ALLOWED_HOSTS = [get_pod_ip(), "horizon.horizon-kuttl-tests.svc"]
  File "/usr/lib/python3.9/site-packages/openstack_dashboard/local/local_settings.py", line 71, in get_pod_ip
    port = 443 if true else 80
NameError: name 'true' is not defined

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: bshephar
Once this PR has been reviewed and has the lgtm label, please assign mcgonago for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[bshephar]

Alright, looks good now from a Kuttl test perspective. @deshipu @mcgonago Let me know if you have any concerns with the change here.

[softwarefactory-project-zuul]

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/6f10386c0b8d4d79952c45901c95f010

❌ openstack-k8s-operators-content-provider TIMED_OUT in 30m 50s
⚠️ horizon-operator-kuttl SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider (non-voting)

[bshephar]

/test horizon-operator-build-deploy-kuttl

[bshephar]

recheck

[stuggi]

@deshipu @mcgonago can you please review this?

[bshephar]

Hey @deshipu and @mcgonago. Happy New Year folks, I hope you had a nice break.

I know @deshipu wasn't a huge fan of this implementation to begin with. I would be keen to hear from your experience if you can think of a better way to address the initial problem that this function was added to address. To give you the quick overview of the problem.

We don't want to just allow anything in ALLOWED_HOSTS, so we start by adding the endpoint address. This allows Horizon to accept incoming requests destined for the Horizon Route. Sweet! Now Horizon can serve user traffic coming from outside of Kubernetes. That's great, works well, exactly what we need.

But, we ALSO want Kubernetes to be able to confirm that each pod is functioning and serving traffic. So, if we just rely on healthchecks via the route, or the Kubernetes Service. Then, k8s services (with ovn-kubernetes) are defined as OVN LoadBalancers that send traffic to all of the pods defined for the loadbalancers backend. Not ideal, since we could never be sure of which pod was just checked. So Kubernetes is unable to reliably determine if the pod we just started is actually working. If we try to check the pod directly, then we can't because the IP address isn't in ALLOWED_HOSTS, so we get blocked.

To solve for this, I wrote the function in this file and used it to add the pods IP address to ALLOWED_HOSTS. Then the healthcheck is configured to use the HTTPGetAction:
https://github.com/openstack-k8s-operators/horizon-operator/blob/main/internal/horizon/deployment.go#L205
https://pkg.go.dev/k8s.io/api/core/v1#HTTPGetAction

How about this as an alternative?
#550