[WIP] cache-fixes

[perdasilva]

Description of the change:

Motivation for the change:

Architectural changes:

Testing remarks:

Reviewer Checklist

• Implementation matches the proposed design, or proposal is updated to match implementation
• Sufficient unit test coverage
• Sufficient end-to-end test coverage
• Bug fixes are accompanied by regression test(s)
• e2e tests and flake fixes are accompanied evidence of flake testing, e.g. executing the test 100(0) times
• tech debt/todo is accompanied by issue link(s) in comments in the surrounding code
• Tests are comprehensible, e.g. Ginkgo DSL is being used appropriately
• Docs updated or added to /doc
• Commit messages sensible and descriptive
• Tests marked as [FLAKE] are truly flaky and have an issue
• Code is properly formatted

[stevekuznetsov]

No client using LIST + WATCH should ever have to do this, FWIW.

[perdasilva]

Right, watch events are at least once right?
This mishmash of cached and live requests is biting us in the butt....

[perdasilva]

In case you're curious: the issue is we have a situation in which the namespaces gets nuked and immediately recreated with the subscription, and we're ending up with unsat...

[stevekuznetsov]

Are you using UID checks to detect deletion + recreation? Do you care that it deleted & recreated, or no?

[perdasilva]

in the end the RC was: the typed queues were of any type and we were enqueuing two different types of objects (string and ResourceEvent). Two semantically equal events come in and get sync'ed concurrently because we break dedupe =P

[stevekuznetsov]

Using a Deployment or anything other than a Pod is likely the better approach that won't cause grief in the future.

[perdasilva]

Using a Deployment or anything other than a Pod is likely the better approach that won't cause grief in the future.

Sorry - the PR description was misleading because I fumbled the commit (--amend). What I'm trying to understand is this: #3010