Set nullptr to the listener ptr