Improve __copy_to_user and __copy_from_user performance
authorpopcornmix <popcornmix@gmail.com>
Mon, 28 Nov 2016 16:50:04 +0000 (16:50 +0000)
committerDom Cobley <popcornmix@gmail.com>
Mon, 21 Mar 2022 16:03:50 +0000 (16:03 +0000)
commit3a3349e8c52c0954467b2d30dd454573d22786b0
tree04d8bf2623aa68f70e26e630b5f21152bf140590
parentf1c4e2f6f4e681926923500c95ef18517fe897ba
Improve __copy_to_user and __copy_from_user performance

Provide a __copy_from_user that uses memcpy. On BCM2708, use
optimised memcpy/memmove/memcmp/memset implementations.

arch/arm: Add mmiocpy/set aliases for memcpy/set

See: https://github.com/raspberrypi/linux/issues/1082

copy_from_user: CPU_SW_DOMAIN_PAN compatibility

The downstream copy_from_user acceleration must also play nice with
CONFIG_CPU_SW_DOMAIN_PAN.

See: https://github.com/raspberrypi/linux/issues/1381

Signed-off-by: Phil Elwell <phil@raspberrypi.org>
Fix copy_from_user if BCM2835_FAST_MEMCPY=n

The change which introduced CONFIG_BCM2835_FAST_MEMCPY unconditionally
changed the behaviour of arm_copy_from_user. The page pinning code
is not safe on ARMv7 if LPAE & high memory is enabled and causes
crashes which look like PTE corruption.

Make __copy_from_user_memcpy conditional on CONFIG_2835_FAST_MEMCPY=y
which is really an ARMv6 / Pi1 optimization and not necessary on newer
ARM processors.

arm: fix mmap unlocks in uaccess_with_memcpy.c

This is a regression that was added with the commit 192a4e923ef092924dd013e7326f2ec520ee4783 as of rpi-5.8.y, since that is when the move to the mmap locking API was introduced - d8ed45c5dcd455fc5848d47f86883a1b872ac0d0

The issue is that when the patch to improve performance for the __copy_to_user and __copy_from_user functions were added for the Raspberry Pi, some of the mmaps were incorrectly mapped to write instead of read. This would cause a verity of issues, and in my case, prevent the booting of a squashfs filesystem on rpi-5.8-y and above. An example of the panic you would see from this can be seen at https://pastebin.com/raw/jBz5xCzL

Signed-off-by: Christian Lamparter <chunkeey@gmail.com>
Signed-off-by: Christopher Blake <chrisrblake93@gmail.com>
13 files changed:
arch/arm/include/asm/string.h
arch/arm/include/asm/uaccess.h
arch/arm/lib/Makefile
arch/arm/lib/arm-mem.h [new file with mode: 0644]
arch/arm/lib/copy_from_user.S
arch/arm/lib/exports_rpi.c [new file with mode: 0644]
arch/arm/lib/memcmp_rpi.S [new file with mode: 0644]
arch/arm/lib/memcpy_rpi.S [new file with mode: 0644]
arch/arm/lib/memcpymove.h [new file with mode: 0644]
arch/arm/lib/memmove_rpi.S [new file with mode: 0644]
arch/arm/lib/memset_rpi.S [new file with mode: 0644]
arch/arm/lib/uaccess_with_memcpy.c
arch/arm/mach-bcm/Kconfig