It seems like the constructed delete queries only match the IDs, while with morph relations you'd have to check both the ID & type columns. This leads to unwanted deletes.
I realized that the hard way, maybe it would be worth documenting this limitation and/or skipping the data (like you seem to do with belongsToMany) until this type of relation is supported.