mm: change lookup_node() to use get_user_pages_fast()

The purpose of calling get_user_pages_locked() from lookup_node() was to
allow for unlocking the mmap_lock when reading a page from the disk
during a page fault (hidden behind VM_FAULT_RETRY).  The idea was to
reduce contention on the heavily-used mmap_lock.  (Thanks to Jan Kara
for clearly pointing that out, and in fact I've used some of his wording
here.)

However, it is unlikely for lookup_node() to take a page fault.  With
that in mind, change over to calling get_user_pages_fast().  This
simplifies the code, runs a little faster in the expected case, and
allows removing get_user_pages_locked() entirely, in a subsequent patch.

Link: https://lkml.kernel.org/r/20220204020010.68930-5-jhubbard@nvidia.com
Signed-off-by: John Hubbard <jhubbard@nvidia.com>
Reviewed-by: Jan Kara <jack@suse.cz>
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Reviewed-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Cc: Alex Williamson <alex.williamson@redhat.com>
Cc: Andrea Arcangeli <aarcange@redhat.com>
Cc: David Hildenbrand <david@redhat.com>
Cc: Jason Gunthorpe <jgg@ziepe.ca>
Cc: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Cc: Lukas Bulwahn <lukas.bulwahn@gmail.com>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Cc: Peter Xu <peterx@redhat.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index 69284d3b5e53f868c77da3ecd32aa3a436e8d359..340c22c6219336166900e6f1d9674dd5985ed546 100644 (file)
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -907,17 +907,14 @@ static void get_policy_nodemask(struct mempolicy *p, nodemask_t *nodes)
 static int lookup_node(struct mm_struct *mm, unsigned long addr)
 {
        struct page *p = NULL;
-       int err;
+       int ret;
 
-       int locked = 1;
-       err = get_user_pages_locked(addr & PAGE_MASK, 1, 0, &p, &locked);
-       if (err > 0) {
-               err = page_to_nid(p);
+       ret = get_user_pages_fast(addr & PAGE_MASK, 1, 0, &p);
+       if (ret > 0) {
+               ret = page_to_nid(p);
                put_page(p);
        }
-       if (locked)
-               mmap_read_unlock(mm);
-       return err;
+       return ret;
 }
 
 /* Retrieve NUMA policy */
@@ -968,14 +965,14 @@ static long do_get_mempolicy(int *policy, nodemask_t *nmask,
        if (flags & MPOL_F_NODE) {
                if (flags & MPOL_F_ADDR) {
                        /*
-                        * Take a refcount on the mpol, lookup_node()
-                        * will drop the mmap_lock, so after calling
-                        * lookup_node() only "pol" remains valid, "vma"
-                        * is stale.
+                        * Take a refcount on the mpol, because we are about to
+                        * drop the mmap_lock, after which only "pol" remains
+                        * valid, "vma" is stale.
                         */
                        pol_refcount = pol;
                        vma = NULL;
                        mpol_get(pol);
+                       mmap_read_unlock(mm);
                        err = lookup_node(mm, addr);
                        if (err < 0)
                                goto out;