i2c: imx: Make sure to unregister adapter on remove()