Thanks to @liuyuan10 for spotting that there a scenario which is leading to presence of duplicate routes in the pod's namespace:
fd36::3:0:e:0:0/96 dev gpu0rdma2 proto kernel metric 256 pref medium
fd36::3:0:e:0:0/96 dev gpu0rdma2 metric 1024 pref medium
The behavior is only observed for interfaces with an IPv6 address that represents a subnet (rather than a complete /128). In this case, the kernel likely automatically creates a route for the IPv6 address that represents the subnet, e.g. the /96 above