Maybe use unregister_taxonomy_for_object_type() function
This can be a way to unregister a built-in tax for the current CPT
This need some test to check how to work with existing CPT like "post"
Also, care about local storage of PostType and maybe do a diff between this and the get_object_taxonomies() ( again, care with hook priority if we can do something )