Fix label-related logic during permission setting