io_uring: gate iowait schedule on having pending requests

A previous commit made all cqring waits marked as iowait, as a way to
improve performance for short schedules with pending IO. However, for
use cases that have a special reaper thread that does nothing but
wait on events on the ring, this causes a cosmetic issue where we
know have one core marked as being "busy" with 100% iowait.

While this isn't a grave issue, it is confusing to users. Rather than
always mark us as being in iowait, gate setting of current->in_iowait
to 1 by whether or not the waiting task has pending requests.

Cc: stable@vger.kernel.org
Link: https://lore.kernel.org/io-uring/CAMEGJJ2RxopfNQ7GNLhr7X9=bHXKo+G5OOe0LUq=+UgLXsv1Xg@mail.gmail.com/
Link: https://bugzilla.kernel.org/show_bug.cgi?id=217699
Link: https://bugzilla.kernel.org/show_bug.cgi?id=217700
Reported-by: Oleksandr Natalenko <oleksandr@natalenko.name>
Reported-by: Phil Elwell <phil@raspberrypi.com>
Tested-by: Andres Freund <andres@anarazel.de>
Fixes: 8a796565cec3 ("io_uring: Use io_schedule* in cqring wait")
Signed-off-by: Jens Axboe <axboe@kernel.dk>

diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c
index 89a6115..f4591b9 100644 (file)
--- a/io_uring/io_uring.c
+++ b/io_uring/io_uring.c
@@ -2493,11 +2493,20 @@ int io_run_task_work_sig(struct io_ring_ctx *ctx)
        return 0;
 }
 
+static bool current_pending_io(void)
+{
+       struct io_uring_task *tctx = current->io_uring;
+
+       if (!tctx)
+               return false;
+       return percpu_counter_read_positive(&tctx->inflight);
+}
+
 /* when returns >0, the caller should retry */
 static inline int io_cqring_wait_schedule(struct io_ring_ctx *ctx,
                                          struct io_wait_queue *iowq)
 {
-       int token, ret;
+       int io_wait, ret;
 
        if (unlikely(READ_ONCE(ctx->check_cq)))
                return 1;
@@ -2511,17 +2520,19 @@ static inline int io_cqring_wait_schedule(struct io_ring_ctx *ctx,
                return 0;
 
        /*
-        * Use io_schedule_prepare/finish, so cpufreq can take into account
-        * that the task is waiting for IO - turns out to be important for low
-        * QD IO.
+        * Mark us as being in io_wait if we have pending requests, so cpufreq
+        * can take into account that the task is waiting for IO - turns out
+        * to be important for low QD IO.
         */
-       token = io_schedule_prepare();
+       io_wait = current->in_iowait;
+       if (current_pending_io())
+               current->in_iowait = 1;
        ret = 0;
        if (iowq->timeout == KTIME_MAX)
                schedule();
        else if (!schedule_hrtimeout(&iowq->timeout, HRTIMER_MODE_ABS))
                ret = -ETIME;
-       io_schedule_finish(token);
+       current->in_iowait = io_wait;
        return ret;
 }