asim: ensure consistent sequencing of changes applied to a range

Upon investigating, we identified the non-deterministic behavior was due to the
sequence of changes applied on the ranges within one tick. Changes within a tick
are applied in one go. But the changes are stored in a map, and map iteration
order is non-deterministic. Thus, the sequence of events applied and published
via gossip could vary and could result in different event states. This patch
resolves the issue by changing the storage of changes to be applied from a map
to an array, ensuring consistent iteration and sequence of events. Note the
shift to an array doesn't pose any issues since all changes need to be applied
anyway.

With this fix, the test `TestAsimDeterministic` fails only under race. This is
less concerning since the simulator should run on a single goroutine without
concurrency. Although the cause of the test failure in race conditions in
unclear, fixing this is not a priority.

Part Of: cockroachdb#105904
Release Note: None

Author: wenyihu6

pkg/kv/kvserver/asim/state/change.go

@@ -477,25 +477,25 @@ func (rc *replicaChanger) Push(tick time.Time, change Change) (time.Time, bool)
 // Tick updates state changer to apply any changes that have occurred
 // between the last tick and this one.
 func (rc *replicaChanger) Tick(tick time.Time, state State) {
-	changeList := make(map[int]*pendingChange)
+	var changeList []*pendingChange
 
 	// NB: Add the smallest unit of time, in order to find all items in
 	// [smallest, tick].
 	pivot := &pendingChange{completeAt: tick.Add(time.Nanosecond)}
 	rc.completeAt.AscendLessThan(pivot, func(i btree.Item) bool {
 		nextChange, _ := i.(*pendingChange)
-		changeList[nextChange.ticket] = nextChange
+		changeList = append(changeList, nextChange)
 		return true
 	})
 
-	for ticket, nextChange := range changeList {
+	for _, nextChange := range changeList {
 		change := rc.pendingTickets[nextChange.ticket]
 		change.Apply(state)
 
 		// Cleanup the pending trackers for this ticket. This allows another
 		// change to be pushed for Range().
 		rc.completeAt.Delete(nextChange)
-		delete(rc.pendingTickets, ticket)
+		delete(rc.pendingTickets, nextChange.ticket)
 		delete(rc.pendingRange, change.Range())
 	}
 }