As a way to break the dependency on compilers, I am suggesting having a way to generate all of the contents used by the autodoc and then have a switch that would either read those contents (for RTD building), or use the current approach (for local development).
The generated files here could either be rst/md files, or just a json file to be a lookaside cache.
I suggest the interface being a sphinx builder because it can then easily slot in people's CI system along with linkcheck and others, but having it always generated is also an option.