maintenance mode feature flag gets disabled on disc in 3.10

[gomoripeti]

Reproduced on RabbitMQ 3.10.8 Erlang 24.3.4.3 (which was upgraded in many steps from 3.7.16)

After the below steps joining a node to another node, the maintenance_mode_status feature flag is temporarily detected as disabled and removed from the feature_flags file, while in memory it stays enabled throughout.
Steps I did:

• create 1 node cluster with node-01
• create node-02 and join it to node-01
• remove node-01 from cluster and delete all data
• create new node-01 from scratch and join it to node-02

In the last step when the new node-01 joins node-02, that is when the feature flag gets removed from the file. I suspect the reason is that the migration function for maintenance_mode_status_migration with arg: is_enabled is run when either whole Mnesia is temporarily stopped or it is reset and the maintenance_mode table is not created yet. (maybe the boot step to create hasn't run yet)

The problem with this inconsistent status is that upgrading to 3.11 would fail.
It cannot be fixed by rabbitmqctl enable_feature_flag maintenance_mode_status because in memory it is already enabled.

This is not applicable for 3.11+ as maintenance mode FF is mandatory there. Also feature_flags_v2 and khepri are coming. But this still looks like a problem on a supported version. And might highlight some more general ordering issue with how feature flags are bootstrapped and checked. For example I'm not sure why the flag is not disabled also in memory (in the generated module)?

Some relevant logs from between stop_app and start_app on new node-01
feature_flag_maint_mode_disabled.log

This is a very similar issue to rabbitmq/discussions#183 and #3704, but there was an improvement included in 3.10.1 to address them.