MySQL K8s fails unit address resolution when pod IP has multiple PTR records

[gboutry]

Steps to reproduce

1. Deploy the MySQL K8s charm on Kubernetes.
2. Let a non-primary unit resolve its unit address while DNS still exposes multiple PTR records for the pod IP.
3. Observe get_unit_address() retrying until it raises.

Expected behavior

The charm should resolve the Kubernetes canonical hostname for the unit reliably, even when multiple PTR records exist for the same pod IP.

Actual behavior

get_unit_address() retries and then raises because the DNS name returned is not the expected unit endpoint name:

File "/var/lib/juju/agents/unit-mysql-1/charm/src/charm.py", line 378, in get_unit_address
    raise RuntimeError("unit DNS domain name is not fully propagated yet")
RuntimeError: unit DNS domain name is not fully propagated yet

The affected unit repeatedly logs:

unit-mysql-1: 07:18:44 WARNING unit.mysql/1.juju-log get_unit_address: unit DNS domain name is not fully propagated yet, trying again
unit-mysql-1: 07:19:06 WARNING unit.mysql/1.juju-log get_unit_address: unit DNS domain name is not fully propagated yet, trying again
unit-mysql-1: 07:19:08 WARNING unit.mysql/1.juju-log get_unit_address: unit DNS domain name is not fully propagated yet, trying again
unit-mysql-1: 07:19:10 WARNING unit.mysql/1.juju-log get_unit_address: unit DNS domain name is not fully propagated yet, trying again
unit-mysql-1: 07:19:12 WARNING unit.mysql/1.juju-log get_unit_address: unit DNS domain name is not fully propagated yet, trying again

Reverse lookups show multiple PTR records for the same IPs, including service and endpoint names:

root@nova-0:/# nslookup 10.1.1.210
;; Got recursion not available from 10.152.183.70
210.1.1.10.in-addr.arpa    name = 10-1-1-210.mysql.openstack.svc.cluster.local.
210.1.1.10.in-addr.arpa    name = mysql-0.mysql-endpoints.openstack.svc.cluster.local.
210.1.1.10.in-addr.arpa    name = 10-1-1-210.mysql-primary.openstack.svc.cluster.local.

root@nova-0:/# nslookup 10.1.0.143
;; Got recursion not available from 10.152.183.70
143.0.1.10.in-addr.arpa    name = 10-1-0-143.mysql.openstack.svc.cluster.local.
143.0.1.10.in-addr.arpa    name = mysql-1.mysql-endpoints.openstack.svc.cluster.local.
143.0.1.10.in-addr.arpa    name = 10-1-0-143.mysql-replicas.openstack.svc.cluster.local.

root@nova-0:/# nslookup 10.1.2.155
;; Got recursion not available from 10.152.183.70
155.2.1.10.in-addr.arpa    name = 10-1-2-155.mysql.openstack.svc.cluster.local.
155.2.1.10.in-addr.arpa    name = mysql-2.mysql-endpoints.openstack.svc.cluster.local.
155.2.1.10.in-addr.arpa    name = 10-1-2-155.mysql-replicas.openstack.svc.cluster.local.

Versions

Operating system: Not provided
Juju CLI: Not provided
Juju agent: Not provided
Charm revision: 346 (mysql 8.0/stable)
Microk8s: Not provided
LXD: Not applicable

Log output

Juju debug log: Not attached

Relevant log excerpts:

File "/var/lib/juju/agents/unit-mysql-1/charm/src/charm.py", line 378, in get_unit_address
    raise RuntimeError("unit DNS domain name is not fully propagated yet")
RuntimeError: unit DNS domain name is not fully propagated yet

unit-mysql-1: 07:18:44 WARNING unit.mysql/1.juju-log get_unit_address: unit DNS domain name is not fully propagated yet, trying again
unit-mysql-1: 07:19:06 WARNING unit.mysql/1.juju-log get_unit_address: unit DNS domain name is not fully propagated yet, trying again
unit-mysql-1: 07:19:08 WARNING unit.mysql/1.juju-log get_unit_address: unit DNS domain name is not fully propagated yet, trying again
unit-mysql-1: 07:19:10 WARNING unit.mysql/1.juju-log get_unit_address: unit DNS domain name is not fully propagated yet, trying again
unit-mysql-1: 07:19:12 WARNING unit.mysql/1.juju-log get_unit_address: unit DNS domain name is not fully propagated yet, trying again

Additional context

The failure appears to come from relying on reverse DNS / getfqdn() semantics when Kubernetes DNS returns multiple PTR records for a pod IP. A canonical-name lookup via getaddrinfo(..., AI_CANONNAME) appears to be more reliable for this case.
https://bugs.python.org/issue5004

[syncronize-issues-to-jira]

Thank you for reporting your feedback to us!

The internal ticket has been created: https://warthogs.atlassian.net/browse/DPE-9572.

This message was autogenerated

[astrojuanlu]

Thank you @gboutry for the bug report and the fix. The extra context from the CPython issue tracker is super useful.

I've seen this frequently in logs as well and indeed, it tends to fix itself. Yet making the logic more robust seems like a no brainer.

[astrojuanlu]

Fixed in 8.0/edge by #138, pending backport to 8.4/edge

[swetha1654]

Observation (just FYI) @astrojuanlu

I investigated this issue further to see if the libraries in traefik are also affected (since they also use socket.getfqn), and realised this is affecting mysql charm only is because the charm is trying to retrieve the hostname of its peer units as well, whereas we are trying to retrieve the localhost fdqn.
Looking at the /etc/nsswitch.conf inside the container, it is first looking at /etc/hosts and then going to the DNS if the record is not present.
So getfqdn is consistent over time locally, whereas for peer hostnames, it reaches DNS and produces inconsistent results.

I tested this inside the container as well. I ran socket.getfdqn() on an infinite loop with the local hostname for 5 mins, there was no change, but that wasn't the case for the peer unit hostname.