io_uring: convert iopoll_completed to store_release
authorPavel Begunkov <asml.silence@gmail.com>
Fri, 24 Sep 2021 20:59:51 +0000 (21:59 +0100)
committerJens Axboe <axboe@kernel.dk>
Tue, 19 Oct 2021 11:49:53 +0000 (05:49 -0600)
commitb3fa03fd1b17f557e2c43736440feed66fb86741
tree9aa3acc5cf662945a9755ac699285189e5a41c94
parent3aa83bfb6e5c3a89d0ec67e598ee04bfc1425b13
io_uring: convert iopoll_completed to store_release

Convert explicit barrier around iopoll_completed to smp_load_acquire()
and smp_store_release(). Similar on the callback side, but replaces a
single smp_rmb() with per-request smp_load_acquire(), neither imply any
extra CPU ordering for x86. Use READ_ONCE as usual where it doesn't
matter.

Use it to move filling CQEs by iopoll earlier, that will be necessary
to avoid traversing the list one extra time in the future.

Suggested-by: Bart Van Assche <bvanassche@acm.org>
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/8bd663cb15efdc72d6247c38ee810964e744a450.1632516769.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
fs/io_uring.c