locking/percpu-rwsem: Optimize readers and reduce global impact
authorPeter Zijlstra <peterz@infradead.org>
Thu, 14 Jul 2016 18:08:46 +0000 (20:08 +0200)
committerIngo Molnar <mingo@kernel.org>
Wed, 10 Aug 2016 12:34:01 +0000 (14:34 +0200)
commit80127a39681bd68c959f0953f84a830cbd7c3b1c
tree223bcc2a5cbec5c0873f8fae85a98797f94e6c56
parent08be8f63c40c030b5cf95b4368e314e563a86301
locking/percpu-rwsem: Optimize readers and reduce global impact

Currently the percpu-rwsem switches to (global) atomic ops while a
writer is waiting; which could be quite a while and slows down
releasing the readers.

This patch cures this problem by ordering the reader-state vs
reader-count (see the comments in __percpu_down_read() and
percpu_down_write()). This changes a global atomic op into a full
memory barrier, which doesn't have the global cacheline contention.

This also enables using the percpu-rwsem with rcu_sync disabled in order
to bias the implementation differently, reducing the writer latency by
adding some cost to readers.

Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Oleg Nesterov <oleg@redhat.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Paul McKenney <paulmck@linux.vnet.ibm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: linux-kernel@vger.kernel.org
[ Fixed modular build. ]
Signed-off-by: Ingo Molnar <mingo@kernel.org>
include/linux/percpu-rwsem.h
kernel/locking/percpu-rwsem.c
kernel/rcu/sync.c