[stable31] fix(conversation): Only trigger event with users that really got removed

[stable31] fix(conversation): Only trigger event with users that really got removed #5499