InspectorPrimitive seems initially like such a flexible solution.
Sadly, the entirety of bevy_inspector cant be used, since its functions take &mut World and can't take &mut RestrictedWorldView. This means that its far harder to write custom UI than it needs to be.
Related, some of the useful functions used by built-in InspectorPrimitive implementations are private: