spi: cadence: revert "Add SPI transfer delays"