mm: store compound_nr as well as compound_order

Patch series "THP prep patches".

These are some generic cleanups and improvements, which I would like
merged into mmotm soon.  The first one should be a performance improvement
for all users of compound pages, and the others are aimed at getting code
to compile away when CONFIG_TRANSPARENT_HUGEPAGE is disabled (ie small
systems).  Also better documented / less confusing than the current prefix
mixture of compound, hpage and thp.

This patch (of 7):

This removes a few instructions from functions which need to know how many
pages are in a compound page.  The storage used is either page->mapping on
64-bit or page->index on 32-bit.  Both of these are fine to overlay on
tail pages.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: William Kucharski <william.kucharski@oracle.com>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Cc: David Hildenbrand <david@redhat.com>
Cc: Mike Kravetz <mike.kravetz@oracle.com>
Cc: "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com>
Link: http://lkml.kernel.org/r/20200629151959.15779-1-willy@infradead.org
Link: http://lkml.kernel.org/r/20200629151959.15779-2-willy@infradead.org
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

diff --git a/include/linux/mm.h b/include/linux/mm.h
index e7602a3bcef1b1030f8049540863914812688bb3..10b1e6e5f8855d77b1c72c61747db16902487e40 100644 (file)
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -922,12 +922,15 @@ static inline int compound_pincount(struct page *page)
 static inline void set_compound_order(struct page *page, unsigned int order)
 {
        page[1].compound_order = order;
+       page[1].compound_nr = 1U << order;
 }
 
 /* Returns the number of pages in this potentially compound page. */
 static inline unsigned long compound_nr(struct page *page)
 {
-       return 1UL << compound_order(page);
+       if (!PageHead(page))
+               return 1;
+       return page[1].compound_nr;
 }
 
 /* Returns the number of bytes in this potentially compound page. */

diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
index 0277fbab7c93bb387ecda083c9d41554668a8fac..496c3ff97cce7abea90e18f4a6cced9be0847769 100644 (file)
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -134,6 +134,7 @@ struct page {
                        unsigned char compound_dtor;
                        unsigned char compound_order;
                        atomic_t compound_mapcount;
+                       unsigned int compound_nr; /* 1 << compound_order */
                };
                struct {        /* Second tail page of compound page */
                        unsigned long _compound_pad_1;  /* compound_head */

diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 8b7d0ecf30b1c3c876aea7ada40c999772b1a6f7..0e2bab486fea6f4d2a6f79c13255d33e097dcb9e 100644 (file)
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -666,8 +666,6 @@ void prep_compound_page(struct page *page, unsigned int order)
        int i;
        int nr_pages = 1 << order;
 
-       set_compound_page_dtor(page, COMPOUND_PAGE_DTOR);
-       set_compound_order(page, order);
        __SetPageHead(page);
        for (i = 1; i < nr_pages; i++) {
                struct page *p = page + i;
@@ -675,6 +673,9 @@ void prep_compound_page(struct page *page, unsigned int order)
                p->mapping = TAIL_MAPPING;
                set_compound_head(p, page);
        }
+
+       set_compound_page_dtor(page, COMPOUND_PAGE_DTOR);
+       set_compound_order(page, order);
        atomic_set(compound_mapcount_ptr(page), -1);
        if (hpage_pincount_available(page))
                atomic_set(compound_pincount_ptr(page), 0);