Fix cleanup logic regarding notify fd

No match.