arm64: entry.S: Do not preempt from IRQ before all cpufeatures are enabled
authorJulien Thierry <julien.thierry@arm.com>
Tue, 15 Oct 2019 17:25:44 +0000 (18:25 +0100)
committerWill Deacon <will@kernel.org>
Wed, 16 Oct 2019 16:51:43 +0000 (09:51 -0700)
commit19c95f261c6558d4c2cbbfacd2d8bb6501384601
tree6876704c309f4249006edae5c9cab054b6111f02
parent8c551f919a73c1dfa690a70a691be1da394145e8
arm64: entry.S: Do not preempt from IRQ before all cpufeatures are enabled

Preempting from IRQ-return means that the task has its PSTATE saved
on the stack, which will get restored when the task is resumed and does
the actual IRQ return.

However, enabling some CPU features requires modifying the PSTATE. This
means that, if a task was scheduled out during an IRQ-return before all
CPU features are enabled, the task might restore a PSTATE that does not
include the feature enablement changes once scheduled back in.

* Task 1:

PAN == 0 ---|                          |---------------
            |                          |<- return from IRQ, PSTATE.PAN = 0
            | <- IRQ                   |
            +--------+ <- preempt()  +--
                                     ^
                                     |
                                     reschedule Task 1, PSTATE.PAN == 1
* Init:
        --------------------+------------------------
                            ^
                            |
                            enable_cpu_features
                            set PSTATE.PAN on all CPUs

Worse than this, since PSTATE is untouched when task switching is done,
a task missing the new bits in PSTATE might affect another task, if both
do direct calls to schedule() (outside of IRQ/exception contexts).

Fix this by preventing preemption on IRQ-return until features are
enabled on all CPUs.

This way the only PSTATE values that are saved on the stack are from
synchronous exceptions. These are expected to be fatal this early, the
exception is BRK for WARN_ON(), but as this uses do_debug_exception()
which keeps IRQs masked, it shouldn't call schedule().

Signed-off-by: Julien Thierry <julien.thierry@arm.com>
[james: Replaced a really cool hack, with an even simpler static key in C.
 expanded commit message with Julien's cover-letter ascii art]
Signed-off-by: James Morse <james.morse@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
arch/arm64/kernel/entry.S
arch/arm64/kernel/process.c
include/linux/sched.h