io_uring: improve cqe !tracing hot path
authorPavel Begunkov <asml.silence@gmail.com>
Thu, 24 Aug 2023 22:53:23 +0000 (23:53 +0100)
committerJens Axboe <axboe@kernel.dk>
Thu, 24 Aug 2023 23:16:19 +0000 (17:16 -0600)
commita0727c738309a06ef5579c1742f8f0def63aa883
tree2733d843c0313e5259bbb8948f5e14ace262d729
parent04d9244c9420db33149608a566399176d57690f8
io_uring: improve cqe !tracing hot path

While looking at io_fill_cqe_req()'s asm I stumbled on our trace points
turning into the chunk below:

trace_io_uring_complete(req->ctx, req, req->cqe.user_data,
req->cqe.res, req->cqe.flags,
req->extra1, req->extra2);

io_uring/io_uring.c:898:  trace_io_uring_complete(req->ctx, req, req->cqe.user_data,
movq 232(%rbx), %rdi # req_44(D)->big_cqe.extra2, _5
movq 224(%rbx), %rdx # req_44(D)->big_cqe.extra1, _6
movl 84(%rbx), %r9d # req_44(D)->cqe.D.81184.flags, _7
movl 80(%rbx), %r8d # req_44(D)->cqe.res, _8
movq 72(%rbx), %rcx # req_44(D)->cqe.user_data, _9
movq 88(%rbx), %rsi # req_44(D)->ctx, _10
./arch/x86/include/asm/jump_label.h:27:  asm_volatile_goto("1:"
1:jmp .L1772 # objtool NOPs this  #
...

It does a jump_label for actual tracing, but those 6 moves will stay
there in the hottest io_uring path. As an optimisation, add a
trace_io_uring_complete_enabled() check, which is also uses jump_labels,
it tricks the compiler into behaving. It removes the junk without
changing anything else int the hot path.

Note: apparently, it's not only me noticing it, and people are also
working it around. We should remove the check when it's solved
generically or rework tracing.

Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/555d8312644b3776f4be7e23f9b92943875c4bc7.1692916914.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
io_uring/io_uring.h