Don't use live client to find what isn't in the cache

[perdasilva]

          n.b. this is not a safe thing to do, as you no longer operate against a consistent view of cluster state that existed in one point in time (your lister is behind - but you got an event for this configmap? how?)

Originally posted by @stevekuznetsov in #3414 (comment)

[github-actions]

Issues go stale after 90 days of inactivity. If there is no further activity, the issue will be closed in another 30 days.