When a pod with a hostPort is moved from hostNetwork: false to hostNetwork: true the plugin is called with an empty RuntimeConfig.PortMaps this is expected because the pod is in the host's network namespace and will not need NAT rule. 108c2ae handles the case where the pod is replaced with a new IP by flushing conntrack the add command.
I think the solution is for the del command to also flush conntrack.