gpiolib: i2c/spi drivers: handle rmmod better