Canonical way to set flags from a custom CMake toolchain without interfering with vcpkg

[msftradford]

I have a custom toolchain defined that sets the following CMake variables:
CMAKE_CXX_COMPILER
CMAKE_CXX_FLAGS
CMAKE_EXE_LINKER_FLAGS

This works fine for our own internal build, but when building vcpkg dependencies it causes issues because it has polluted the variables:
CMAKE_CXX_FLAGS
CMAKE_EXE_LINKER_FLAGS

It seems like I need to make sure that the flags for our internal build are not used in vcpkg and the flags for vcpkg are not used in the internal build. Further, it seems like the only thing that is shared is the compiler.

What is the canonical way to approach this problem? I can think of a couple of paths forward but none of them seem to be all that clean and this seems like a common enough problem that others would have solved it. Thanks for any advice!