USB: serial: drop if with an always false condition

USB: serial: drop if with an always false condition

In a bus remove function the passed device is always valid, so there is
no need to check for it being NULL.

(Side note: The check for port being non-NULL is broken anyhow, because
to_usb_serial_port() is a wrapper around container_of() for a member that is
not the first one. So port can hardly become NULL.)

Signed-off-by: Uwe Kleine-König <uwe@kleine-koenig.org>
Link: https://lore.kernel.org/r/20210208143149.963644-1-uwe@kleine-koenig.org
Signed-off-by: Johan Hovold <johan@kernel.org>