xdp: improve page_pool xdp_return performance
authorJesper Dangaard Brouer <brouer@redhat.com>
Wed, 21 Sep 2022 17:05:32 +0000 (19:05 +0200)
committerJakub Kicinski <kuba@kernel.org>
Mon, 26 Sep 2022 18:28:19 +0000 (11:28 -0700)
commitfb33ec016b8710281343ce73bec92bfe54bad4fa
treef471e8b3f7980468c82d5651c112fa620bbb160c
parentc931b060f0939efc2e4c0eeaa5ae8268a7c5c22c
xdp: improve page_pool xdp_return performance

During LPC2022 I meetup with my page_pool co-maintainer Ilias. When
discussing page_pool code we realised/remembered certain optimizations
had not been fully utilised.

Since commit c07aea3ef4d4 ("mm: add a signature in struct page") struct
page have a direct pointer to the page_pool object this page was
allocated from.

Thus, with this info it is possible to skip the rhashtable_lookup to
find the page_pool object in __xdp_return().

The rcu_read_lock can be removed as it was tied to xdp_mem_allocator.
The page_pool object is still safe to access as it tracks inflight pages
and (potentially) schedules final release from a work queue.

Created a micro benchmark of XDP redirecting from mlx5 into veth with
XDP_DROP bpf-prog on the peer veth device. This increased performance
6.5% from approx 8.45Mpps to 9Mpps corresponding to using 7 nanosec
(27 cycles at 3.8GHz) less per packet.

Suggested-by: Ilias Apalodimas <ilias.apalodimas@linaro.org>
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
Acked-by: Toke Høiland-Jørgensen <toke@redhat.com>
Reviewed-by: Ilias Apalodimas <ilias.apalodimas@linaro.org>
Link: https://lore.kernel.org/r/166377993287.1737053.10258297257583703949.stgit@firesoul
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
net/core/xdp.c