If the converter is running out of options, it could look for a public static op_Implicit or op_Explicit method on the source type that returns the destination type and accepts a single parameter of the source type, then call it.
See also: https://stackoverflow.com/a/32025393
The reflection result should be cached, and it should be possible to clear the cache on demand to free memory.