PBM-1665 keep PITR running and prioritize different node for profile backups

[jcechace]

Ticket: https://perconadev.atlassian.net/browse/PBM-1665

PITR was being stopped only temporarily – when backup arrived, the main PITR slicer would be signaled, take a last slice and return. However the PITR monitor rutine in Agent would then check active backup logs and in case of profile storage, start the PITR anew. This took about 15s

This PR adjust the behavior in two ways

1. It doesn't signal the PITR slicer if the backup is not on main storage. So it never get stopped for those 15s
2. If backup is not on the main storage, it checks what node is currently running PITR and penalizes its priority heavily (unless that would interfere with prior priority adjustments). This way the backup will preferably start on a different node.

[boris-ilijic]

Looks good, but it's necessary to expand it from use-case perspective.

[boris-ilijic]

This will not work correctly for PSA. From requirements:

In case of PSA topology, prioritize Secondary.

P: 1.0; S (after deprioritize): 0.1 --> backup will be executed on P, and it should go on S.

[jcechace]

P is actually 1/2, but your point is valid anyway. The priority for PITR secondary should be > 1/2. Since the number matters, are you OK if we make the score constants in pnm/prio/priority.go public?

[boris-ilijic]

deprioritizePITRNodes deals with coefficient not scores, so maybe we can make that intention clear by setting coefficient prio.defaultScore - 0.1. That would be well good rule for protecting PITR node of running backup.

I would remove fixed 0.6, it's kind "where's this number coming from?" :)
But all good for me, this would work now.

[jcechace]

@boris-ilijic this should be do the trick for PSA. But I would prefer making the score constants in priority package public and use that here.