mm/page_alloc: track range of active PCP lists during bulk free

free_pcppages_bulk() frees pages in a round-robin fashion.  Originally,
this was dealing only with migratetypes but storing high-order pages
means that there can be many more empty lists that are uselessly
checked.  Track the minimum and maximum active pindex to reduce the
search space.

Link: https://lkml.kernel.org/r/20220217002227.5739-3-mgorman@techsingularity.net
Signed-off-by: Mel Gorman <mgorman@techsingularity.net>
Reviewed-by: Vlastimil Babka <vbabka@suse.cz>
Tested-by: Aaron Lu <aaron.lu@intel.com>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Jesper Dangaard Brouer <brouer@redhat.com>
Cc: Michal Hocko <mhocko@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 35c399d..63edba3 100644 (file)
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -1447,6 +1447,8 @@ static void free_pcppages_bulk(struct zone *zone, int count,
                                        struct per_cpu_pages *pcp)
 {
        int pindex = 0;
+       int min_pindex = 0;
+       int max_pindex = NR_PCP_LISTS - 1;
        int batch_free = 0;
        int nr_freed = 0;
        unsigned int order;
@@ -1472,13 +1474,20 @@ static void free_pcppages_bulk(struct zone *zone, int count,
                 */
                do {
                        batch_free++;
-                       if (++pindex == NR_PCP_LISTS)
-                               pindex = 0;
+                       if (++pindex > max_pindex)
+                               pindex = min_pindex;
                        list = &pcp->lists[pindex];
-               } while (list_empty(list));
+                       if (!list_empty(list))
+                               break;
+
+                       if (pindex == max_pindex)
+                               max_pindex--;
+                       if (pindex == min_pindex)
+                               min_pindex++;
+               } while (1);
 
                /* This is the only non-empty list. Free them all. */
-               if (batch_free == NR_PCP_LISTS)
+               if (batch_free >= max_pindex - min_pindex)
                        batch_free = count;
 
                order = pindex_to_order(pindex);