[Improvement] Prioritize reconciliation of new work

[sudheerv]

Describe the bug

A clear and concise description of what the bug is.

apply_controllers in the memberagent currently watches all the Work resources created in the designated namespace on the Hub cluster and reconciles them in order. This can result in serious bottleneck in applying newer resources or newer updates to existing resources over time especially when there are 1000s of namespaces created a long time ago and never have any updates to them, yet reconciled one by one before the controller can get to the newer ones.

Environment

Please provide the following:

• Hub cluster details
• Member cluster details
  Single hub cluster serving 10-100 member clusters. Hubagent and memberagent both running on newer AMD machines with enough cores and memory. Fleet is configured to propagate namespace and rbac resources created in the Hub to memberclusters.

To Reproduce

Steps to reproduce the behavior:
Create a new namespace on the hub that already has 5000 namespaces. It takes 30-45 minutes for the propagation and for marking ClusterResourcePlacement ready.

Expected behavior

A clear and concise description of what you expected to happen.
Newer resources should be propagated within a handful of minutes of creating

Screenshots

If applicable, add screenshots to help explain your problem.

Additional context

Add any other context about the problem here.