[stable30] fix(conversation): Only trigger event with users that really got removed

[stable30] fix(conversation): Only trigger event with users that really got removed #16772