The way that it was envisioned is that 'removing' someone from a trust relation, is done by moving the trust relationship to a next state (next trust identifier) in which the shard holder that needs to be removed is excluded from.
So shard holders don't loose 'access' to the trust identifier/relationship it was in, but the trust identifier moved on to a next state in which that shard holder is not a part of anymore.