[Fix] Remove restriction from tree when user initiated remove operation