spmi: Add a check for remove callback when removing a SPMI driver