mmc: sdhci_am654: Properly handle failures in .remove()

mmc: sdhci_am654: Properly handle failures in .remove()

Returning an error code in a platform driver's remove function is wrong
most of the time and there is an effort to make the callback return
void. To prepare this rework the function not to exit early.

Cc: Uwe Kleine-König <u.kleine-koenig@pengutronix.de>
Signed-off-by: Yangtao Li <frank.li@vivo.com>
Acked-by: Adrian Hunter <adrian.hunter@intel.com>
Link: https://lore.kernel.org/r/20230727070051.17778-59-frank.li@vivo.com
Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>