We should make sure we have (at minimum) testing on the behavior described in https://github.com/OSC/ondemand/pull/5019#issue-3861702104