mm: cma: allocate cma areas bottom-up

Currently cma areas without a fixed base are allocated close to the end of
the node.  This placement is sub-optimal because of compaction: it brings
pages into the cma area.  In particular, it can bring in hot executable
pages, even if there is a plenty of free memory on the machine.  This
results in cma allocation failures.

Instead let's place cma areas close to the beginning of a node.  In this
case the compaction will help to free cma areas, resulting in better cma
allocation success rates.

If there is enough memory let's try to allocate bottom-up starting with
4GB to exclude any possible interference with DMA32.  On smaller machines
or in a case of a failure, stick with the old behavior.

16GB vm, 2GB cma area:
With this patch:
[    0.000000] Command line: root=/dev/vda3 rootflags=subvol=/root systemd.unified_cgroup_hierarchy=1 enforcing=0 console=ttyS0,115200 hugetlb_cma=2G
[    0.002928] hugetlb_cma: reserve 2048 MiB, up to 2048 MiB per node
[    0.002930] cma: Reserved 2048 MiB at 0x0000000100000000
[    0.002931] hugetlb_cma: reserved 2048 MiB on node 0

Without this patch:
[    0.000000] Command line: root=/dev/vda3 rootflags=subvol=/root systemd.unified_cgroup_hierarchy=1 enforcing=0 console=ttyS0,115200 hugetlb_cma=2G
[    0.002930] hugetlb_cma: reserve 2048 MiB, up to 2048 MiB per node
[    0.002933] cma: Reserved 2048 MiB at 0x00000003c0000000
[    0.002934] hugetlb_cma: reserved 2048 MiB on node 0

v2:
  - switched to memblock_set_bottom_up(true), by Mike
  - start with 4GB, by Mike

[guro@fb.com: whitespace fix, per Mike]
Link: https://lkml.kernel.org/r/20201221170551.GB3428478@carbon.DHCP.thefacebook.com
[guro@fb.com: fix 32-bit warnings]
Link: https://lkml.kernel.org/r/20201223163537.GA4011967@carbon.DHCP.thefacebook.com
[guro@fb.com: fix 32-bit systems]
[akpm@linux-foundation.org: build fix]

Link: https://lkml.kernel.org/r/20201217201214.3414100-1-guro@fb.com
Signed-off-by: Roman Gushchin <guro@fb.com>
Reviewed-by: Mike Rapoport <rppt@linux.ibm.com>
Cc: Wonhyuk Yang <vvghjk1234@gmail.com>
Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com>
Cc: Rik van Riel <riel@surriel.com>
Cc: Michal Hocko <mhocko@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

diff --git a/mm/cma.c b/mm/cma.c
index 20c4f6f40037d8ecc7bcd0f7c35fe0048d117a72..0ba69cd16aeb5dc6807da2450c833d64959e0ce1 100644 (file)
--- a/mm/cma.c
+++ b/mm/cma.c
@@ -336,6 +336,23 @@ int __init cma_declare_contiguous_nid(phys_addr_t base,
                        limit = highmem_start;
                }
 
+               /*
+                * If there is enough memory, try a bottom-up allocation first.
+                * It will place the new cma area close to the start of the node
+                * and guarantee that the compaction is moving pages out of the
+                * cma area and not into it.
+                * Avoid using first 4GB to not interfere with constrained zones
+                * like DMA/DMA32.
+                */
+#ifdef CONFIG_PHYS_ADDR_T_64BIT
+               if (!memblock_bottom_up() && memblock_end >= SZ_4G + size) {
+                       memblock_set_bottom_up(true);
+                       addr = memblock_alloc_range_nid(size, alignment, SZ_4G,
+                                                       limit, nid, true);
+                       memblock_set_bottom_up(false);
+               }
+#endif
+
                if (!addr) {
                        addr = memblock_alloc_range_nid(size, alignment, base,
                                        limit, nid, true);