ie implement the change at https://github.com/lightning/bolts/pull/1014 we also need to keep track of channels we removed so we don't immediately re-add them.