io_uring: optimise submission side poll_refs
authorPavel Begunkov <asml.silence@gmail.com>
Thu, 23 Jun 2022 13:24:49 +0000 (14:24 +0100)
committerJens Axboe <axboe@kernel.dk>
Mon, 25 Jul 2022 00:39:16 +0000 (18:39 -0600)
commit49f1c68e048f1706b71c8255faf8110113d1cc48
tree2c187dab428dded372ad655f7e4ce627797fd8f4
parentde08356f4858628fdffb8bd7e9cceb60c7e08ead
io_uring: optimise submission side poll_refs

The final poll_refs put in __io_arm_poll_handler() takes quite some
cycles. When we're arming from the original task context task_work won't
be run, so in this case we can assume that we won't race with task_works
and so not take the initial ownership ref.

One caveat is that after arming a poll we may race with it, so we have
to add a bunch of io_poll_get_ownership() hidden inside of
io_poll_can_finish_inline() whenever we want to complete arming inline.
For the same reason we can't just set REQ_F_DOUBLE_POLL in
__io_queue_proc() and so need to sync with the first poll entry by
taking its wq head lock.

Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/8825315d7f5e182ac1578a031e546f79b1c97d01.1655990418.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
io_uring/poll.c