bpf: Enable non-atomic allocations in local storage
authorJoanne Koong <joannelkoong@gmail.com>
Fri, 18 Mar 2022 04:55:52 +0000 (21:55 -0700)
committerAlexei Starovoitov <ast@kernel.org>
Mon, 21 Mar 2022 01:55:05 +0000 (18:55 -0700)
commitb00fa38a9c1cba044a32a601b49a55a18ed719d1
tree871c74ded225f4bd7108f611a7a8e86f1642e16c
parenta8fee96202e279441d0e52d83eb100bd4a6d6272
bpf: Enable non-atomic allocations in local storage

Currently, local storage memory can only be allocated atomically
(GFP_ATOMIC). This restriction is too strict for sleepable bpf
programs.

In this patch, the verifier detects whether the program is sleepable,
and passes the corresponding GFP_KERNEL or GFP_ATOMIC flag as a
5th argument to bpf_task/sk/inode_storage_get. This flag will propagate
down to the local storage functions that allocate memory.

Please note that bpf_task/sk/inode_storage_update_elem functions are
invoked by userspace applications through syscalls. Preemption is
disabled before bpf_task/sk/inode_storage_update_elem is called, which
means they will always have to allocate memory atomically.

Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Acked-by: KP Singh <kpsingh@kernel.org>
Acked-by: Martin KaFai Lau <kafai@fb.com>
Link: https://lore.kernel.org/bpf/20220318045553.3091807-2-joannekoong@fb.com
include/linux/bpf_local_storage.h
kernel/bpf/bpf_inode_storage.c
kernel/bpf/bpf_local_storage.c
kernel/bpf/bpf_task_storage.c
kernel/bpf/verifier.c
net/core/bpf_sk_storage.c