The directions for adding a Custom VM differ from the directions from a normal CA rotation in that the custom CA skips all the required steps of rebuilding all VMs before enabling the new CA. This (I think) will lead to foundation failure.
Custom CA Step 2 (just enable it!) - https://docs.pivotal.io/pivotalcf/2-3/security/pcf-infrastructure/custom-ca-cert.html#add
Normal CA Rotation - has steps 4, 5, 6, 7, 8, and 9 that I think are also needed for a Custom CA rotation - https://docs.pivotal.io/pivotalcf/2-3/security/pcf-infrastructure/api-cert-rotation.html
If the intent was to apply a custom CA BEFORE deploying PCF then I believe the Custom CA steps will work, after installation, it will not.