Different VAE for Base, Hires, & Refiner

[Quidam2k]

I'm trying to set up a workflow where I do the base generation using PonyXL, then a refiner that's a 1.5 model, and also want to use hires fix, and, if it matters, adetailer.

I've gone in and set the Preferred VAE for the models I'm using, and tried messing around with the checkboxes and such in the VAE section of the Settings, but I'm still getting VAE glitching when it runs the refiner.

I've tried searching around, and found a potential solution that uses Comfy, but I'd prefer to stick with Forge if I can, and to avoid having to run img2img on them in post.

Am I missing an obvious step? Has anyone got a workflow that does this?