mm: slub: add missing TID bump in kmem_cache_alloc_bulk()
authorJann Horn <jannh@google.com>
Tue, 17 Mar 2020 00:28:45 +0000 (01:28 +0100)
committerGreg Kroah-Hartman <gregkh@linuxfoundation.org>
Fri, 20 Mar 2020 08:07:58 +0000 (09:07 +0100)
commitff58bb340c430da021875283f8bf251630c61183
tree08dad33b29d53902a2e29fba3910678bb0b8285d
parented14ef08c55ad6ebe94d2cadf373817d326239c2
mm: slub: add missing TID bump in kmem_cache_alloc_bulk()

commit fd4d9c7d0c71866ec0c2825189ebd2ce35bd95b8 upstream.

When kmem_cache_alloc_bulk() attempts to allocate N objects from a percpu
freelist of length M, and N > M > 0, it will first remove the M elements
from the percpu freelist, then call ___slab_alloc() to allocate the next
element and repopulate the percpu freelist. ___slab_alloc() can re-enable
IRQs via allocate_slab(), so the TID must be bumped before ___slab_alloc()
to properly commit the freelist head change.

Fix it by unconditionally bumping c->tid when entering the slowpath.

Cc: stable@vger.kernel.org
Fixes: ebe909e0fdb3 ("slub: improve bulk alloc strategy")
Signed-off-by: Jann Horn <jannh@google.com>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
mm/slub.c