[9.0] Fixes testSnapshotShutdownProgressTracker (#134926)

[joshua-adams-1]

Backports #134926 to 9.0

I was able to reproduce the error by following the steps outlined by @JeremyDahlgren here

The error occurs due to the following sequence of events:

1. One of the snapshot threads has a sleep when executing snapshot tasks
2. The other is not delayed, so it will get blocked in the repository to allow the test to get to the putShutdownForRemovalMetadata() call
3. After the delays the first thread processes the other snapshot tasks, which are now paused, which throws a PausedSnapshotException, and then immediately notifies the master with the failure
4. The first four SnapshotShutdownProgressTracker log messages are verified in the test and then the masterTransportService.addRequestHandlingBehavior() is called to block the notifications
5. The repo is unblocked on the nodeForRemoval, but by this time it is too late and one or more updates have already been sent, possibly leaving only the single blocked shard, causing the assertion at 581 to fail

The proposed change:

1. Ensures that there are n shards on otherNode with data. At the moment, there is only 1.
2. I have reordered the blocking on the master node to come earlier. This is so we are always capturing the notifications back to the master, either by the snapshot thread being blocked, or if there was some delay on a thread due to other concurrent scheduling.

A full explanation of the changes can be found here

---

Testing

• Having been able to reproducible make the test fail, it now succeeds 100 times in a row
• The original gradle command ./gradlew ":server:internalClusterTest" --tests "org.elasticsearch.snapshots.SnapshotShutdownIT.testSnapshotShutdownProgressTracker" -Dtests.seed=C44FB8C3BA8AC78E -Dtests.locale=dz -Dtests.timezone=Australia/Queensland -Druntime.java=23 succeeds
• CI passes

Resolves: #134620

[elasticsearchmachine]

Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination)

[JeremyDahlgren]

LGTM