nvmet-rdma: fix use-after-free when a port is removed