spi: Fix tegra20 build with CONFIG_PM=n