bpf: Remove extra lock_sock for TCP_ZEROCOPY_RECEIVE
authorStanislav Fomichev <sdf@google.com>
Fri, 15 Jan 2021 16:34:59 +0000 (08:34 -0800)
committerAlexei Starovoitov <ast@kernel.org>
Wed, 20 Jan 2021 22:23:00 +0000 (14:23 -0800)
commit9cacf81f8161111db25f98e78a7a0e32ae142b3f
treef20ef5d2014e204f3084ebbea54515427fdef43e
parent13ca51d5eb358edcb673afccb48c3440b9fda21b
bpf: Remove extra lock_sock for TCP_ZEROCOPY_RECEIVE

Add custom implementation of getsockopt hook for TCP_ZEROCOPY_RECEIVE.
We skip generic hooks for TCP_ZEROCOPY_RECEIVE and have a custom
call in do_tcp_getsockopt using the on-stack data. This removes
3% overhead for locking/unlocking the socket.

Without this patch:
     3.38%     0.07%  tcp_mmap  [kernel.kallsyms]  [k] __cgroup_bpf_run_filter_getsockopt
            |
             --3.30%--__cgroup_bpf_run_filter_getsockopt
                       |
                        --0.81%--__kmalloc

With the patch applied:
     0.52%     0.12%  tcp_mmap  [kernel.kallsyms]  [k] __cgroup_bpf_run_filter_getsockopt_kern

Note, exporting uapi/tcp.h requires removing netinet/tcp.h
from test_progs.h because those headers have confliciting
definitions.

Signed-off-by: Stanislav Fomichev <sdf@google.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Acked-by: Martin KaFai Lau <kafai@fb.com>
Link: https://lore.kernel.org/bpf/20210115163501.805133-2-sdf@google.com
16 files changed:
include/linux/bpf-cgroup.h
include/linux/indirect_call_wrapper.h
include/net/sock.h
include/net/tcp.h
kernel/bpf/cgroup.c
net/ipv4/tcp.c
net/ipv4/tcp_ipv4.c
net/ipv6/tcp_ipv6.c
net/socket.c
tools/include/uapi/linux/tcp.h [new file with mode: 0644]
tools/testing/selftests/bpf/prog_tests/bpf_tcp_ca.c
tools/testing/selftests/bpf/prog_tests/cls_redirect.c
tools/testing/selftests/bpf/prog_tests/sockmap_basic.c
tools/testing/selftests/bpf/prog_tests/sockopt_sk.c
tools/testing/selftests/bpf/progs/sockopt_sk.c
tools/testing/selftests/bpf/test_progs.h