iochannel: remove fd from poll() when we don't care from events