xhci: Make handover code more robust