Training runs with identical configs (up to refactoring and assuming that default values have not changed, as far as I can tell) between jlp/vision_model and hybrid_dev, on a small training set.
The loss curves look close enough, although the one in hybrid_dev seems slightly lower:

Another question is: should we enforce some parts of the vision-encoder config like causal: false, cross_document_attention: False, rotary.type: default_2d ?