xfs: don't assert fail on perag references on teardown