spl: serial_bcm283x: remove unnecessary gpio setting