iomap: support IOCB_DIO_CALLER_COMP
authorJens Axboe <axboe@kernel.dk>
Sat, 8 Jul 2023 16:01:50 +0000 (10:01 -0600)
committerJens Axboe <axboe@kernel.dk>
Tue, 1 Aug 2023 23:32:49 +0000 (17:32 -0600)
commit8c052fb3002e6e7eccdc138d2e459cd03f019ade
tree34b8f53a5c1db89e6dd17f762c969040c9cc0228
parent099ada2c87260e5c52fbdad4ca40c165fc194a2e
iomap: support IOCB_DIO_CALLER_COMP

If IOCB_DIO_CALLER_COMP is set, utilize that to set kiocb->dio_complete
handler and data for that callback. Rather than punt the completion to a
workqueue, we pass back the handler and data to the issuer and will get
a callback from a safe task context.

Using the following fio job to randomly dio write 4k blocks at
queue depths of 1..16:

fio --name=dio-write --filename=/data1/file --time_based=1 \
--runtime=10 --bs=4096 --rw=randwrite --norandommap --buffered=0 \
--cpus_allowed=4 --ioengine=io_uring --iodepth=$depth

shows the following results before and after this patch:

Stock Patched Diff
=======================================
QD1 155K 162K + 4.5%
QD2 290K 313K + 7.9%
QD4 533K 597K +12.0%
QD8 604K 827K +36.9%
QD16 615K 845K +37.4%

which shows nice wins all around. If we factored in per-IOP efficiency,
the wins look even nicer. This becomes apparent as queue depth rises,
as the offloaded workqueue completions runs out of steam.

Reviewed-by: Darrick J. Wong <djwong@kernel.org>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
fs/iomap/direct-io.c