Remove remained idle callback when destroy handle