Add cancelations to policy monitor dispatches

[michel-laterman]

What is the problem this PR solves?

Add support for canceling dispatching policies to agents. This occurs whenever a new policy is recieved by the policy monitor. This allows the monitor to stop sending revision N when N+1 is recieved.

How does this PR solve the problem?

Policy dispatches now occur in a goroutine. If a new doc is detected by the dispatch, the routine is canceled and another is launched.

Design Checklist

• I have ensured my design is stateless and will work when multiple fleet-server instances are behind a load balancer.
• I have or intend to scale test my changes, ensuring it will work reliably with 100K+ agents connected.
• I have included fail safe mechanisms to limit the load on fleet-server: rate limiting, circuit breakers, caching, load shedding, etc.

Checklist

• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool

Related issues

• Relates Policy monitor load issues #3254

[prodsecmachine]

🎉 Snyk checks have passed. No issues have been found so far.

✅ security/snyk check is complete. No issues have been found. (View Details)

✅ license/snyk check is complete. No issues have been found. (View Details)

[mergify]

This pull request does not have a backport label. Could you fix it @michel-laterman? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-./d./d is the label to automatically backport to the 8./d branch. /d is the digit
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[michel-laterman]

buildkite run perf-tests

[michel-laterman]

buildkite run perf-tests

[blakerouse]

Overall this looks good and is a nice improvement. See inline comment about checking the error type.

[blakerouse]

Looks good.

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
90.3% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube