io_uring: improve sq_thread waiting check

If SQPOLL task finds a ring requesting it to continue running, no need
to set wake flag to rest of the rings as it will be cleared in a moment
anyway, so hide it in a single sqd->ctx_list loop.

Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/1ee5a696d9fd08645994c58ee147d149a8957d94.1621201931.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>

diff --git a/fs/io_uring.c b/fs/io_uring.c
index 24c0042..85d0184 100644 (file)
--- a/fs/io_uring.c
+++ b/fs/io_uring.c
@@ -6894,11 +6894,10 @@ static int io_sq_thread(void *data)
 
                prepare_to_wait(&sqd->wait, &wait, TASK_INTERRUPTIBLE);
                if (!io_sqd_events_pending(sqd)) {
-                       list_for_each_entry(ctx, &sqd->ctx_list, sqd_list)
-                               io_ring_set_wakeup_flag(ctx);
-
                        needs_sched = true;
                        list_for_each_entry(ctx, &sqd->ctx_list, sqd_list) {
+                               io_ring_set_wakeup_flag(ctx);
+
                                if ((ctx->flags & IORING_SETUP_IOPOLL) &&
                                    !list_empty_careful(&ctx->iopoll_list)) {
                                        needs_sched = false;