mm: slab: optimize memcg_slab_free_hook()
authorMuchun Song <songmuchun@bytedance.com>
Fri, 29 Apr 2022 12:30:44 +0000 (20:30 +0800)
committerVlastimil Babka <vbabka@suse.cz>
Mon, 4 Jul 2022 15:13:05 +0000 (17:13 +0200)
commitb77d5b1b83e3e14870224de7c63f115a2dc44e9a
tree4c9ad0b977e29f88a0867df5002db0a1e938821c
parentb347aa7b57477f71c740e2bbc6d1078a7109ba23
mm: slab: optimize memcg_slab_free_hook()

Most callers of memcg_slab_free_hook() already know the slab,  which could
be passed to memcg_slab_free_hook() directly to reduce the overhead of an
another call of virt_to_slab().  For bulk freeing of objects, the call of
slab_objcgs() in the loop in memcg_slab_free_hook() is redundant as well.
Rework memcg_slab_free_hook() and build_detached_freelist() to reduce
those unnecessary overhead and make memcg_slab_free_hook() can handle bulk
freeing in slab_free().

Move the calling site of memcg_slab_free_hook() from do_slab_free() to
slab_free() for slub to make the code clearer since the logic is weird
(e.g. the caller need to judge whether it needs to call
memcg_slab_free_hook()). It is easy to make mistakes like missing calling
of memcg_slab_free_hook() like fixes of:

  commit d1b2cf6cb84a ("mm: memcg/slab: uncharge during kmem_cache_free_bulk()")
  commit ae085d7f9365 ("mm: kfence: fix missing objcg housekeeping for SLAB")

This optimization is mainly for bulk objects freeing.  The following numbers
is shown for 16-object freeing.

                           before      after
  kmem_cache_free_bulk:   ~430 ns     ~400 ns

The overhead is reduced by about 7% for 16-object freeing.

Signed-off-by: Muchun Song <songmuchun@bytedance.com>
Reviewed-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
Link: https://lore.kernel.org/r/20220429123044.37885-1-songmuchun@bytedance.com
Signed-off-by: Vlastimil Babka <vbabka@suse.cz>
mm/slab.c
mm/slab.h
mm/slub.c