io_uring: cache task struct refs
authorPavel Begunkov <asml.silence@gmail.com>
Mon, 14 Jun 2021 01:36:22 +0000 (02:36 +0100)
committerJens Axboe <axboe@kernel.dk>
Mon, 14 Jun 2021 14:23:13 +0000 (08:23 -0600)
commit09899b19155a152f3ff4eb5c203232175d630fbc
treece5d7f987b8c9a9bc1f0002a014048e813a19f5b
parent2d091d62b1106e90f195599c67bf385ddedfc915
io_uring: cache task struct refs

tctx in submission part is always synchronised because is executed from
the task's context, so we can batch allocate tctx/task references and
store them across syscall boundaries. It avoids enough of operations,
including an atomic for getting task ref and a percpu_counter_add()
function call, which still fallback to spinlock for large batching
cases (around >=32). Should be good for SQPOLL submitting in small
portions and coming at some moment bpf submissions.

Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/14b327b973410a3eec1f702ecf650e100513aca9.1623634181.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
fs/io_uring.c