summaryrefslogtreecommitdiffstats
diff options
context:
space:
mode:
authorMel Gorman <mgorman@techsingularity.net>2016-07-28 18:45:24 -0400
committerLinus Torvalds <torvalds@linux-foundation.org>2016-07-28 19:07:41 -0400
commit75ef7184053989118d3814c558a9af62e7376a58 (patch)
tree4e343fb8be9c58d7ffb96076968c866c7d64ea11
parenta621184ac6dd415664eae458ed1727ba235f5450 (diff)
mm, vmstat: add infrastructure for per-node vmstats
Patchset: "Move LRU page reclaim from zones to nodes v9" This series moves LRUs from the zones to the node. While this is a current rebase, the test results were based on mmotm as of June 23rd. Conceptually, this series is simple but there are a lot of details. Some of the broad motivations for this are; 1. The residency of a page partially depends on what zone the page was allocated from. This is partially combatted by the fair zone allocation policy but that is a partial solution that introduces overhead in the page allocator paths. 2. Currently, reclaim on node 0 behaves slightly different to node 1. For example, direct reclaim scans in zonelist order and reclaims even if the zone is over the high watermark regardless of the age of pages in that LRU. Kswapd on the other hand starts reclaim on the highest unbalanced zone. A difference in distribution of file/anon pages due to when they were allocated results can result in a difference in again. While the fair zone allocation policy mitigates some of the problems here, the page reclaim results on a multi-zone node will always be different to a single-zone node. it was scheduled on as a result. 3. kswapd and the page allocator scan zones in the opposite order to avoid interfering with each other but it's sensitive to timing. This mitigates the page allocator using pages that were allocated very recently in the ideal case but it's sensitive to timing. When kswapd is allocating from lower zones then it's great but during the rebalancing of the highest zone, the page allocator and kswapd interfere with each other. It's worse if the highest zone is small and difficult to balance. 4. slab shrinkers are node-based which makes it harder to identify the exact relationship between slab reclaim and LRU reclaim. The reason we have zone-based reclaim is that we used to have large highmem zones in common configurations and it was necessary to quickly find ZONE_NORMAL pages for reclaim. Today, this is much less of a concern as machines with lots of memory will (or should) use 64-bit kernels. Combinations of 32-bit hardware and 64-bit hardware are rare. Machines that do use highmem should have relatively low highmem:lowmem ratios than we worried about in the past. Conceptually, moving to node LRUs should be easier to understand. The page allocator plays fewer tricks to game reclaim and reclaim behaves similarly on all nodes. The series has been tested on a 16 core UMA machine and a 2-socket 48 core NUMA machine. The UMA results are presented in most cases as the NUMA machine behaved similarly. pagealloc --------- This is a microbenchmark that shows the benefit of removing the fair zone allocation policy. It was tested uip to order-4 but only orders 0 and 1 are shown as the other orders were comparable. 4.7.0-rc4 4.7.0-rc4 mmotm-20160623 nodelru-v9 Min total-odr0-1 490.00 ( 0.00%) 457.00 ( 6.73%) Min total-odr0-2 347.00 ( 0.00%) 329.00 ( 5.19%) Min total-odr0-4 288.00 ( 0.00%) 273.00 ( 5.21%) Min total-odr0-8 251.00 ( 0.00%) 239.00 ( 4.78%) Min total-odr0-16 234.00 ( 0.00%) 222.00 ( 5.13%) Min total-odr0-32 223.00 ( 0.00%) 211.00 ( 5.38%) Min total-odr0-64 217.00 ( 0.00%) 208.00 ( 4.15%) Min total-odr0-128 214.00 ( 0.00%) 204.00 ( 4.67%) Min total-odr0-256 250.00 ( 0.00%) 230.00 ( 8.00%) Min total-odr0-512 271.00 ( 0.00%) 269.00 ( 0.74%) Min total-odr0-1024 291.00 ( 0.00%) 282.00 ( 3.09%) Min total-odr0-2048 303.00 ( 0.00%) 296.00 ( 2.31%) Min total-odr0-4096 311.00 ( 0.00%) 309.00 ( 0.64%) Min total-odr0-8192 316.00 ( 0.00%) 314.00 ( 0.63%) Min total-odr0-16384 317.00 ( 0.00%) 315.00 ( 0.63%) Min total-odr1-1 742.00 ( 0.00%) 712.00 ( 4.04%) Min total-odr1-2 562.00 ( 0.00%) 530.00 ( 5.69%) Min total-odr1-4 457.00 ( 0.00%) 433.00 ( 5.25%) Min total-odr1-8 411.00 ( 0.00%) 381.00 ( 7.30%) Min total-odr1-16 381.00 ( 0.00%) 356.00 ( 6.56%) Min total-odr1-32 372.00 ( 0.00%) 346.00 ( 6.99%) Min total-odr1-64 372.00 ( 0.00%) 343.00 ( 7.80%) Min total-odr1-128 375.00 ( 0.00%) 351.00 ( 6.40%) Min total-odr1-256 379.00 ( 0.00%) 351.00 ( 7.39%) Min total-odr1-512 385.00 ( 0.00%) 355.00 ( 7.79%) Min total-odr1-1024 386.00 ( 0.00%) 358.00 ( 7.25%) Min total-odr1-2048 390.00 ( 0.00%) 362.00 ( 7.18%) Min total-odr1-4096 390.00 ( 0.00%) 362.00 ( 7.18%) Min total-odr1-8192 388.00 ( 0.00%) 363.00 ( 6.44%) This shows a steady improvement throughout. The primary benefit is from reduced system CPU usage which is obvious from the overall times; 4.7.0-rc4 4.7.0-rc4 mmotm-20160623nodelru-v8 User 189.19 191.80 System 2604.45 2533.56 Elapsed 2855.30 2786.39 The vmstats also showed that the fair zone allocation policy was definitely removed as can be seen here; 4.7.0-rc3 4.7.0-rc3 mmotm-20160623 nodelru-v8 DMA32 allocs 28794729769 0 Normal allocs 48432501431 77227309877 Movable allocs 0 0 tiobench on ext4 ---------------- tiobench is a benchmark that artifically benefits if old pages remain resident while new pages get reclaimed. The fair zone allocation policy mitigates this problem so pages age fairly. While the benchmark has problems, it is important that tiobench performance remains constant as it implies that page aging problems that the fair zone allocation policy fixes are not re-introduced. 4.7.0-rc4 4.7.0-rc4 mmotm-20160623 nodelru-v9 Min PotentialReadSpeed 89.65 ( 0.00%) 90.21 ( 0.62%) Min SeqRead-MB/sec-1 82.68 ( 0.00%) 82.01 ( -0.81%) Min SeqRead-MB/sec-2 72.76 ( 0.00%) 72.07 ( -0.95%) Min SeqRead-MB/sec-4 75.13 ( 0.00%) 74.92 ( -0.28%) Min SeqRead-MB/sec-8 64.91 ( 0.00%) 65.19 ( 0.43%) Min SeqRead-MB/sec-16 62.24 ( 0.00%) 62.22 ( -0.03%) Min RandRead-MB/sec-1 0.88 ( 0.00%) 0.88 ( 0.00%) Min RandRead-MB/sec-2 0.95 ( 0.00%) 0.92 ( -3.16%) Min RandRead-MB/sec-4 1.43 ( 0.00%) 1.34 ( -6.29%) Min RandRead-MB/sec-8 1.61 ( 0.00%) 1.60 ( -0.62%) Min RandRead-MB/sec-16 1.80 ( 0.00%) 1.90 ( 5.56%) Min SeqWrite-MB/sec-1 76.41 ( 0.00%) 76.85 ( 0.58%) Min SeqWrite-MB/sec-2 74.11 ( 0.00%) 73.54 ( -0.77%) Min SeqWrite-MB/sec-4 80.05 ( 0.00%) 80.13 ( 0.10%) Min SeqWrite-MB/sec-8 72.88 ( 0.00%) 73.20 ( 0.44%) Min SeqWrite-MB/sec-16 75.91 ( 0.00%) 76.44 ( 0.70%) Min RandWrite-MB/sec-1 1.18 ( 0.00%) 1.14 ( -3.39%) Min RandWrite-MB/sec-2 1.02 ( 0.00%) 1.03 ( 0.98%) Min RandWrite-MB/sec-4 1.05 ( 0.00%) 0.98 ( -6.67%) Min RandWrite-MB/sec-8 0.89 ( 0.00%) 0.92 ( 3.37%) Min RandWrite-MB/sec-16 0.92 ( 0.00%) 0.93 ( 1.09%) 4.7.0-rc4 4.7.0-rc4 mmotm-20160623 approx-v9 User 645.72 525.90 System 403.85 331.75 Elapsed 6795.36 6783.67 This shows that the series has little or not impact on tiobench which is desirable and a reduction in system CPU usage. It indicates that the fair zone allocation policy was removed in a manner that didn't reintroduce one class of page aging bug. There were only minor differences in overall reclaim activity 4.7.0-rc4 4.7.0-rc4 mmotm-20160623nodelru-v8 Minor Faults 645838 647465 Major Faults 573 640 Swap Ins 0 0 Swap Outs 0 0 DMA allocs 0 0 DMA32 allocs 46041453 44190646 Normal allocs 78053072 79887245 Movable allocs 0 0 Allocation stalls 24 67 Stall zone DMA 0 0 Stall zone DMA32 0 0 Stall zone Normal 0 2 Stall zone HighMem 0 0 Stall zone Movable 0 65 Direct pages scanned 10969 30609 Kswapd pages scanned 93375144 93492094 Kswapd pages reclaimed 93372243 93489370 Direct pages reclaimed 10969 30609 Kswapd efficiency 99% 99% Kswapd velocity 13741.015 13781.934 Direct efficiency 100% 100% Direct velocity 1.614 4.512 Percentage direct scans 0% 0% kswapd activity was roughly comparable. There were differences in direct reclaim activity but negligible in the context of the overall workload (velocity of 4 pages per second with the patches applied, 1.6 pages per second in the baseline kernel). pgbench read-only large configuration on ext4 --------------------------------------------- pgbench is a database benchmark that can be sensitive to page reclaim decisions. This also checks if removing the fair zone allocation policy is safe pgbench Transactions 4.7.0-rc4 4.7.0-rc4 mmotm-20160623 nodelru-v8 Hmean 1 188.26 ( 0.00%) 189.78 ( 0.81%) Hmean 5 330.66 ( 0.00%) 328.69 ( -0.59%) Hmean 12 370.32 ( 0.00%) 380.72 ( 2.81%) Hmean 21 368.89 ( 0.00%) 369.00 ( 0.03%) Hmean 30 382.14 ( 0.00%) 360.89 ( -5.56%) Hmean 32 428.87 ( 0.00%) 432.96 ( 0.95%) Negligible differences again. As with tiobench, overall reclaim activity was comparable. bonnie++ on ext4 ---------------- No interesting performance difference, negligible differences on reclaim stats. paralleldd on ext4 ------------------ This workload uses varying numbers of dd instances to read large amounts of data from disk. 4.7.0-rc3 4.7.0-rc3 mmotm-20160623 nodelru-v9 Amean Elapsd-1 186.04 ( 0.00%) 189.41 ( -1.82%) Amean Elapsd-3 192.27 ( 0.00%) 191.38 ( 0.46%) Amean Elapsd-5 185.21 ( 0.00%) 182.75 ( 1.33%) Amean Elapsd-7 183.71 ( 0.00%) 182.11 ( 0.87%) Amean Elapsd-12 180.96 ( 0.00%) 181.58 ( -0.35%) Amean Elapsd-16 181.36 ( 0.00%) 183.72 ( -1.30%) 4.7.0-rc4 4.7.0-rc4 mmotm-20160623 nodelru-v9 User 1548.01 1552.44 System 8609.71 8515.08 Elapsed 3587.10 3594.54 There is little or no change in performance but some drop in system CPU usage. 4.7.0-rc3 4.7.0-rc3 mmotm-20160623 nodelru-v9 Minor Faults 362662 367360 Major Faults 1204 1143 Swap Ins 22 0 Swap Outs 2855 1029 DMA allocs 0 0 DMA32 allocs 31409797 28837521 Normal allocs 46611853 49231282 Movable allocs 0 0 Direct pages scanned 0 0 Kswapd pages scanned 40845270 40869088 Kswapd pages reclaimed 40830976 40855294 Direct pages reclaimed 0 0 Kswapd efficiency 99% 99% Kswapd velocity 11386.711 11369.769 Direct efficiency 100% 100% Direct velocity 0.000 0.000 Percentage direct scans 0% 0% Page writes by reclaim 2855 1029 Page writes file 0 0 Page writes anon 2855 1029 Page reclaim immediate 771 1628 Sector Reads 293312636 293536360 Sector Writes 18213568 18186480 Page rescued immediate 0 0 Slabs scanned 128257 132747 Direct inode steals 181 56 Kswapd inode steals 59 1131 It basically shows that kswapd was active at roughly the same rate in both kernels. There was also comparable slab scanning activity and direct reclaim was avoided in both cases. There appears to be a large difference in numbers of inodes reclaimed but the workload has few active inodes and is likely a timing artifact. stutter ------- stutter simulates a simple workload. One part uses a lot of anonymous memory, a second measures mmap latency and a third copies a large file. The primary metric is checking for mmap latency. stutter 4.7.0-rc4 4.7.0-rc4 mmotm-20160623 nodelru-v8 Min mmap 16.6283 ( 0.00%) 13.4258 ( 19.26%) 1st-qrtle mmap 54.7570 ( 0.00%) 34.9121 ( 36.24%) 2nd-qrtle mmap 57.3163 ( 0.00%) 46.1147 ( 19.54%) 3rd-qrtle mmap 58.9976 ( 0.00%) 47.1882 ( 20.02%) Max-90% mmap 59.7433 ( 0.00%) 47.4453 ( 20.58%) Max-93% mmap 60.1298 ( 0.00%) 47.6037 ( 20.83%) Max-95% mmap 73.4112 ( 0.00%) 82.8719 (-12.89%) Max-99% mmap 92.8542 ( 0.00%) 88.8870 ( 4.27%) Max mmap 1440.6569 ( 0.00%) 121.4201 ( 91.57%) Mean mmap 59.3493 ( 0.00%) 42.2991 ( 28.73%) Best99%Mean mmap 57.2121 ( 0.00%) 41.8207 ( 26.90%) Best95%Mean mmap 55.9113 ( 0.00%) 39.9620 ( 28.53%) Best90%Mean mmap 55.6199 ( 0.00%) 39.3124 ( 29.32%) Best50%Mean mmap 53.2183 ( 0.00%) 33.1307 ( 37.75%) Best10%Mean mmap 45.9842 ( 0.00%) 20.4040 ( 55.63%) Best5%Mean mmap 43.2256 ( 0.00%) 17.9654 ( 58.44%) Best1%Mean mmap 32.9388 ( 0.00%) 16.6875 ( 49.34%) This shows a number of improvements with the worst-case outlier greatly improved. Some of the vmstats are interesting 4.7.0-rc4 4.7.0-rc4 mmotm-20160623nodelru-v8 Swap Ins 163 502 Swap Outs 0 0 DMA allocs 0 0 DMA32 allocs 618719206 1381662383 Normal allocs 891235743 564138421 Movable allocs 0 0 Allocation stalls 2603 1 Direct pages scanned 216787 2 Kswapd pages scanned 50719775 41778378 Kswapd pages reclaimed 41541765 41777639 Direct pages reclaimed 209159 0 Kswapd efficiency 81% 99% Kswapd velocity 16859.554 14329.059 Direct efficiency 96% 0% Direct velocity 72.061 0.001 Percentage direct scans 0% 0% Page writes by reclaim 6215049 0 Page writes file 6215049 0 Page writes anon 0 0 Page reclaim immediate 70673 90 Sector Reads 81940800 81680456 Sector Writes 100158984 98816036 Page rescued immediate 0 0 Slabs scanned 1366954 22683 While this is not guaranteed in all cases, this particular test showed a large reduction in direct reclaim activity. It's also worth noting that no page writes were issued from reclaim context. This series is not without its hazards. There are at least three areas that I'm concerned with even though I could not reproduce any problems in that area. 1. Reclaim/compaction is going to be affected because the amount of reclaim is no longer targetted at a specific zone. Compaction works on a per-zone basis so there is no guarantee that reclaiming a few THP's worth page pages will have a positive impact on compaction success rates. 2. The Slab/LRU reclaim ratio is affected because the frequency the shrinkers are called is now different. This may or may not be a problem but if it is, it'll be because shrinkers are not called enough and some balancing is required. 3. The anon/file reclaim ratio may be affected. Pages about to be dirtied are distributed between zones and the fair zone allocation policy used to do something very similar for anon. The distribution is now different but not necessarily in any way that matters but it's still worth bearing in mind. VM statistic counters for reclaim decisions are zone-based. If the kernel is to reclaim on a per-node basis then we need to track per-node statistics but there is no infrastructure for that. The most notable change is that the old node_page_state is renamed to sum_zone_node_page_state. The new node_page_state takes a pglist_data and uses per-node stats but none exist yet. There is some renaming such as vm_stat to vm_zone_stat and the addition of vm_node_stat and the renaming of mod_state to mod_zone_state. Otherwise, this is mostly a mechanical patch with no functional change. There is a lot of similarity between the node and zone helpers which is unfortunate but there was no obvious way of reusing the code and maintaining type safety. Link: http://lkml.kernel.org/r/1467970510-21195-2-git-send-email-mgorman@techsingularity.net Signed-off-by: Mel Gorman <mgorman@techsingularity.net> Acked-by: Johannes Weiner <hannes@cmpxchg.org> Acked-by: Vlastimil Babka <vbabka@suse.cz> Cc: Rik van Riel <riel@surriel.com> Cc: Vlastimil Babka <vbabka@suse.cz> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Minchan Kim <minchan@kernel.org> Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com> Cc: Hillf Danton <hillf.zj@alibaba-inc.com> Cc: Michal Hocko <mhocko@kernel.org> Cc: Minchan Kim <minchan@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Diffstat
-rw-r--r--drivers/base/node.c76
-rw-r--r--include/linux/mm.h5
-rw-r--r--include/linux/mmzone.h13
-rw-r--r--include/linux/vmstat.h92
-rw-r--r--mm/page_alloc.c10
-rw-r--r--mm/vmstat.c295
-rw-r--r--mm/workingset.c9
7 files changed, 424 insertions, 76 deletions
diff --git a/drivers/base/node.c b/drivers/base/node.c
index 51c7db2c4ee2..0a1b6433a76c 100644
--- a/drivers/base/node.c
+++ b/drivers/base/node.c
@@ -74,16 +74,16 @@ static ssize_t node_read_meminfo(struct device *dev,
74 nid, K(i.totalram), 74 nid, K(i.totalram),
75 nid, K(i.freeram), 75 nid, K(i.freeram),
76 nid, K(i.totalram - i.freeram), 76 nid, K(i.totalram - i.freeram),
77 nid, K(node_page_state(nid, NR_ACTIVE_ANON) + 77 nid, K(sum_zone_node_page_state(nid, NR_ACTIVE_ANON) +
78 node_page_state(nid, NR_ACTIVE_FILE)), 78 sum_zone_node_page_state(nid, NR_ACTIVE_FILE)),
79 nid, K(node_page_state(nid, NR_INACTIVE_ANON) + 79 nid, K(sum_zone_node_page_state(nid, NR_INACTIVE_ANON) +
80 node_page_state(nid, NR_INACTIVE_FILE)), 80 sum_zone_node_page_state(nid, NR_INACTIVE_FILE)),
81 nid, K(node_page_state(nid, NR_ACTIVE_ANON)), 81 nid, K(sum_zone_node_page_state(nid, NR_ACTIVE_ANON)),
82 nid, K(node_page_state(nid, NR_INACTIVE_ANON)), 82 nid, K(sum_zone_node_page_state(nid, NR_INACTIVE_ANON)),
83 nid, K(node_page_state(nid, NR_ACTIVE_FILE)), 83 nid, K(sum_zone_node_page_state(nid, NR_ACTIVE_FILE)),
84 nid, K(node_page_state(nid, NR_INACTIVE_FILE)), 84 nid, K(sum_zone_node_page_state(nid, NR_INACTIVE_FILE)),
85 nid, K(node_page_state(nid, NR_UNEVICTABLE)), 85 nid, K(sum_zone_node_page_state(nid, NR_UNEVICTABLE)),
86 nid, K(node_page_state(nid, NR_MLOCK))); 86 nid, K(sum_zone_node_page_state(nid, NR_MLOCK)));
87 87
88#ifdef CONFIG_HIGHMEM 88#ifdef CONFIG_HIGHMEM
89 n += sprintf(buf + n, 89 n += sprintf(buf + n,
@@ -117,31 +117,31 @@ static ssize_t node_read_meminfo(struct device *dev,
117 "Node %d ShmemPmdMapped: %8lu kB\n" 117 "Node %d ShmemPmdMapped: %8lu kB\n"
118#endif 118#endif
119 , 119 ,
120 nid, K(node_page_state(nid, NR_FILE_DIRTY)), 120 nid, K(sum_zone_node_page_state(nid, NR_FILE_DIRTY)),
121 nid, K(node_page_state(nid, NR_WRITEBACK)), 121 nid, K(sum_zone_node_page_state(nid, NR_WRITEBACK)),
122 nid, K(node_page_state(nid, NR_FILE_PAGES)), 122 nid, K(sum_zone_node_page_state(nid, NR_FILE_PAGES)),
123 nid, K(node_page_state(nid, NR_FILE_MAPPED)), 123 nid, K(sum_zone_node_page_state(nid, NR_FILE_MAPPED)),
124 nid, K(node_page_state(nid, NR_ANON_PAGES)), 124 nid, K(sum_zone_node_page_state(nid, NR_ANON_PAGES)),
125 nid, K(i.sharedram), 125 nid, K(i.sharedram),
126 nid, node_page_state(nid, NR_KERNEL_STACK) * 126 nid, sum_zone_node_page_state(nid, NR_KERNEL_STACK) *
127 THREAD_SIZE / 1024, 127 THREAD_SIZE / 1024,
128 nid, K(node_page_state(nid, NR_PAGETABLE)), 128 nid, K(sum_zone_node_page_state(nid, NR_PAGETABLE)),
129 nid, K(node_page_state(nid, NR_UNSTABLE_NFS)), 129 nid, K(sum_zone_node_page_state(nid, NR_UNSTABLE_NFS)),
130 nid, K(node_page_state(nid, NR_BOUNCE)), 130 nid, K(sum_zone_node_page_state(nid, NR_BOUNCE)),
131 nid, K(node_page_state(nid, NR_WRITEBACK_TEMP)), 131 nid, K(sum_zone_node_page_state(nid, NR_WRITEBACK_TEMP)),
132 nid, K(node_page_state(nid, NR_SLAB_RECLAIMABLE) + 132 nid, K(sum_zone_node_page_state(nid, NR_SLAB_RECLAIMABLE) +
133 node_page_state(nid, NR_SLAB_UNRECLAIMABLE)), 133 sum_zone_node_page_state(nid, NR_SLAB_UNRECLAIMABLE)),
134 nid, K(node_page_state(nid, NR_SLAB_RECLAIMABLE)), 134 nid, K(sum_zone_node_page_state(nid, NR_SLAB_RECLAIMABLE)),
135#ifdef CONFIG_TRANSPARENT_HUGEPAGE 135#ifdef CONFIG_TRANSPARENT_HUGEPAGE
136 nid, K(node_page_state(nid, NR_SLAB_UNRECLAIMABLE)), 136 nid, K(sum_zone_node_page_state(nid, NR_SLAB_UNRECLAIMABLE)),
137 nid, K(node_page_state(nid, NR_ANON_THPS) * 137 nid, K(sum_zone_node_page_state(nid, NR_ANON_THPS) *
138 HPAGE_PMD_NR), 138 HPAGE_PMD_NR),
139 nid, K(node_page_state(nid, NR_SHMEM_THPS) * 139 nid, K(sum_zone_node_page_state(nid, NR_SHMEM_THPS) *
140 HPAGE_PMD_NR), 140 HPAGE_PMD_NR),
141 nid, K(node_page_state(nid, NR_SHMEM_PMDMAPPED) * 141 nid, K(sum_zone_node_page_state(nid, NR_SHMEM_PMDMAPPED) *
142 HPAGE_PMD_NR)); 142 HPAGE_PMD_NR));
143#else 143#else
144 nid, K(node_page_state(nid, NR_SLAB_UNRECLAIMABLE))); 144 nid, K(sum_zone_node_page_state(nid, NR_SLAB_UNRECLAIMABLE)));
145#endif 145#endif
146 n += hugetlb_report_node_meminfo(nid, buf + n); 146 n += hugetlb_report_node_meminfo(nid, buf + n);
147 return n; 147 return n;
@@ -160,12 +160,12 @@ static ssize_t node_read_numastat(struct device *dev,
160 "interleave_hit %lu\n" 160 "interleave_hit %lu\n"
161 "local_node %lu\n" 161 "local_node %lu\n"
162 "other_node %lu\n", 162 "other_node %lu\n",
163 node_page_state(dev->id, NUMA_HIT), 163 sum_zone_node_page_state(dev->id, NUMA_HIT),
164 node_page_state(dev->id, NUMA_MISS), 164 sum_zone_node_page_state(dev->id, NUMA_MISS),
165 node_page_state(dev->id, NUMA_FOREIGN), 165 sum_zone_node_page_state(dev->id, NUMA_FOREIGN),
166 node_page_state(dev->id, NUMA_INTERLEAVE_HIT), 166 sum_zone_node_page_state(dev->id, NUMA_INTERLEAVE_HIT),
167 node_page_state(dev->id, NUMA_LOCAL), 167 sum_zone_node_page_state(dev->id, NUMA_LOCAL),
168 node_page_state(dev->id, NUMA_OTHER)); 168 sum_zone_node_page_state(dev->id, NUMA_OTHER));
169} 169}
170static DEVICE_ATTR(numastat, S_IRUGO, node_read_numastat, NULL); 170static DEVICE_ATTR(numastat, S_IRUGO, node_read_numastat, NULL);
171 171
@@ -173,12 +173,18 @@ static ssize_t node_read_vmstat(struct device *dev,
173 struct device_attribute *attr, char *buf) 173 struct device_attribute *attr, char *buf)
174{ 174{
175 int nid = dev->id; 175 int nid = dev->id;
176 struct pglist_data *pgdat = NODE_DATA(nid);
176 int i; 177 int i;
177 int n = 0; 178 int n = 0;
178 179
179 for (i = 0; i < NR_VM_ZONE_STAT_ITEMS; i++) 180 for (i = 0; i < NR_VM_ZONE_STAT_ITEMS; i++)
180 n += sprintf(buf+n, "%s %lu\n", vmstat_text[i], 181 n += sprintf(buf+n, "%s %lu\n", vmstat_text[i],
181 node_page_state(nid, i)); 182 sum_zone_node_page_state(nid, i));
183
184 for (i = 0; i < NR_VM_NODE_STAT_ITEMS; i++)
185 n += sprintf(buf+n, "%s %lu\n",
186 vmstat_text[i + NR_VM_ZONE_STAT_ITEMS],
187 node_page_state(pgdat, i));
182 188
183 return n; 189 return n;
184} 190}
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 97065e1f0237..08ed53eeedd5 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -933,6 +933,11 @@ static inline struct zone *page_zone(const struct page *page)
933 return &NODE_DATA(page_to_nid(page))->node_zones[page_zonenum(page)]; 933 return &NODE_DATA(page_to_nid(page))->node_zones[page_zonenum(page)];
934} 934}
935 935
936static inline pg_data_t *page_pgdat(const struct page *page)
937{
938 return NODE_DATA(page_to_nid(page));
939}
940
936#ifdef SECTION_IN_PAGE_FLAGS 941#ifdef SECTION_IN_PAGE_FLAGS
937static inline void set_page_section(struct page *page, unsigned long section) 942static inline void set_page_section(struct page *page, unsigned long section)
938{ 943{
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 19425e988bdc..078ecb81e209 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -160,6 +160,10 @@ enum zone_stat_item {
160 NR_FREE_CMA_PAGES, 160 NR_FREE_CMA_PAGES,
161 NR_VM_ZONE_STAT_ITEMS }; 161 NR_VM_ZONE_STAT_ITEMS };
162 162
163enum node_stat_item {
164 NR_VM_NODE_STAT_ITEMS
165};
166
163/* 167/*
164 * We do arithmetic on the LRU lists in various places in the code, 168 * We do arithmetic on the LRU lists in various places in the code,
165 * so it is important to keep the active lists LRU_ACTIVE higher in 169 * so it is important to keep the active lists LRU_ACTIVE higher in
@@ -267,6 +271,11 @@ struct per_cpu_pageset {
267#endif 271#endif
268}; 272};
269 273
274struct per_cpu_nodestat {
275 s8 stat_threshold;
276 s8 vm_node_stat_diff[NR_VM_NODE_STAT_ITEMS];
277};
278
270#endif /* !__GENERATING_BOUNDS.H */ 279#endif /* !__GENERATING_BOUNDS.H */
271 280
272enum zone_type { 281enum zone_type {
@@ -695,6 +704,10 @@ typedef struct pglist_data {
695 struct list_head split_queue; 704 struct list_head split_queue;
696 unsigned long split_queue_len; 705 unsigned long split_queue_len;
697#endif 706#endif
707
708 /* Per-node vmstats */
709 struct per_cpu_nodestat __percpu *per_cpu_nodestats;
710 atomic_long_t vm_stat[NR_VM_NODE_STAT_ITEMS];
698} pg_data_t; 711} pg_data_t;
699 712
700#define node_present_pages(nid) (NODE_DATA(nid)->node_present_pages) 713#define node_present_pages(nid) (NODE_DATA(nid)->node_present_pages)
diff --git a/include/linux/vmstat.h b/include/linux/vmstat.h
index d2da8e053210..d1744aa3ab9c 100644
--- a/include/linux/vmstat.h
+++ b/include/linux/vmstat.h
@@ -106,20 +106,38 @@ static inline void vm_events_fold_cpu(int cpu)
106 zone_idx(zone), delta) 106 zone_idx(zone), delta)
107 107
108/* 108/*
109 * Zone based page accounting with per cpu differentials. 109 * Zone and node-based page accounting with per cpu differentials.
110 */ 110 */
111extern atomic_long_t vm_stat[NR_VM_ZONE_STAT_ITEMS]; 111extern atomic_long_t vm_zone_stat[NR_VM_ZONE_STAT_ITEMS];
112extern atomic_long_t vm_node_stat[NR_VM_NODE_STAT_ITEMS];
112 113
113static inline void zone_page_state_add(long x, struct zone *zone, 114static inline void zone_page_state_add(long x, struct zone *zone,
114 enum zone_stat_item item) 115 enum zone_stat_item item)
115{ 116{
116 atomic_long_add(x, &zone->vm_stat[item]); 117 atomic_long_add(x, &zone->vm_stat[item]);
117 atomic_long_add(x, &vm_stat[item]); 118 atomic_long_add(x, &vm_zone_stat[item]);
119}
120
121static inline void node_page_state_add(long x, struct pglist_data *pgdat,
122 enum node_stat_item item)
123{
124 atomic_long_add(x, &pgdat->vm_stat[item]);
125 atomic_long_add(x, &vm_node_stat[item]);
118} 126}
119 127
120static inline unsigned long global_page_state(enum zone_stat_item item) 128static inline unsigned long global_page_state(enum zone_stat_item item)
121{ 129{
122 long x = atomic_long_read(&vm_stat[item]); 130 long x = atomic_long_read(&vm_zone_stat[item]);
131#ifdef CONFIG_SMP
132 if (x < 0)
133 x = 0;
134#endif
135 return x;
136}
137
138static inline unsigned long global_node_page_state(enum node_stat_item item)
139{
140 long x = atomic_long_read(&vm_node_stat[item]);
123#ifdef CONFIG_SMP 141#ifdef CONFIG_SMP
124 if (x < 0) 142 if (x < 0)
125 x = 0; 143 x = 0;
@@ -161,31 +179,44 @@ static inline unsigned long zone_page_state_snapshot(struct zone *zone,
161} 179}
162 180
163#ifdef CONFIG_NUMA 181#ifdef CONFIG_NUMA
164 182extern unsigned long sum_zone_node_page_state(int node,
165extern unsigned long node_page_state(int node, enum zone_stat_item item); 183 enum zone_stat_item item);
166 184extern unsigned long node_page_state(struct pglist_data *pgdat,
185 enum node_stat_item item);
167#else 186#else
168 187#define sum_zone_node_page_state(node, item) global_page_state(item)
169#define node_page_state(node, item) global_page_state(item) 188#define node_page_state(node, item) global_node_page_state(item)
170
171#endif /* CONFIG_NUMA */ 189#endif /* CONFIG_NUMA */
172 190
173#define add_zone_page_state(__z, __i, __d) mod_zone_page_state(__z, __i, __d) 191#define add_zone_page_state(__z, __i, __d) mod_zone_page_state(__z, __i, __d)
174#define sub_zone_page_state(__z, __i, __d) mod_zone_page_state(__z, __i, -(__d)) 192#define sub_zone_page_state(__z, __i, __d) mod_zone_page_state(__z, __i, -(__d))
193#define add_node_page_state(__p, __i, __d) mod_node_page_state(__p, __i, __d)
194#define sub_node_page_state(__p, __i, __d) mod_node_page_state(__p, __i, -(__d))
175 195
176#ifdef CONFIG_SMP 196#ifdef CONFIG_SMP
177void __mod_zone_page_state(struct zone *, enum zone_stat_item item, long); 197void __mod_zone_page_state(struct zone *, enum zone_stat_item item, long);
178void __inc_zone_page_state(struct page *, enum zone_stat_item); 198void __inc_zone_page_state(struct page *, enum zone_stat_item);
179void __dec_zone_page_state(struct page *, enum zone_stat_item); 199void __dec_zone_page_state(struct page *, enum zone_stat_item);
180 200
201void __mod_node_page_state(struct pglist_data *, enum node_stat_item item, long);
202void __inc_node_page_state(struct page *, enum node_stat_item);
203void __dec_node_page_state(struct page *, enum node_stat_item);
204
181void mod_zone_page_state(struct zone *, enum zone_stat_item, long); 205void mod_zone_page_state(struct zone *, enum zone_stat_item, long);
182void inc_zone_page_state(struct page *, enum zone_stat_item); 206void inc_zone_page_state(struct page *, enum zone_stat_item);
183void dec_zone_page_state(struct page *, enum zone_stat_item); 207void dec_zone_page_state(struct page *, enum zone_stat_item);
184 208
209void mod_node_page_state(struct pglist_data *, enum node_stat_item, long);
210void inc_node_page_state(struct page *, enum node_stat_item);
211void dec_node_page_state(struct page *, enum node_stat_item);
212
185extern void inc_zone_state(struct zone *, enum zone_stat_item); 213extern void inc_zone_state(struct zone *, enum zone_stat_item);
214extern void inc_node_state(struct pglist_data *, enum node_stat_item);
186extern void __inc_zone_state(struct zone *, enum zone_stat_item); 215extern void __inc_zone_state(struct zone *, enum zone_stat_item);
216extern void __inc_node_state(struct pglist_data *, enum node_stat_item);
187extern void dec_zone_state(struct zone *, enum zone_stat_item); 217extern void dec_zone_state(struct zone *, enum zone_stat_item);
188extern void __dec_zone_state(struct zone *, enum zone_stat_item); 218extern void __dec_zone_state(struct zone *, enum zone_stat_item);
219extern void __dec_node_state(struct pglist_data *, enum node_stat_item);
189 220
190void quiet_vmstat(void); 221void quiet_vmstat(void);
191void cpu_vm_stats_fold(int cpu); 222void cpu_vm_stats_fold(int cpu);
@@ -213,16 +244,34 @@ static inline void __mod_zone_page_state(struct zone *zone,
213 zone_page_state_add(delta, zone, item); 244 zone_page_state_add(delta, zone, item);
214} 245}
215 246
247static inline void __mod_node_page_state(struct pglist_data *pgdat,
248 enum node_stat_item item, int delta)
249{
250 node_page_state_add(delta, pgdat, item);
251}
252
216static inline void __inc_zone_state(struct zone *zone, enum zone_stat_item item) 253static inline void __inc_zone_state(struct zone *zone, enum zone_stat_item item)
217{ 254{
218 atomic_long_inc(&zone->vm_stat[item]); 255 atomic_long_inc(&zone->vm_stat[item]);
219 atomic_long_inc(&vm_stat[item]); 256 atomic_long_inc(&vm_zone_stat[item]);
257}
258
259static inline void __inc_node_state(struct pglist_data *pgdat, enum node_stat_item item)
260{
261 atomic_long_inc(&pgdat->vm_stat[item]);
262 atomic_long_inc(&vm_node_stat[item]);
220} 263}
221 264
222static inline void __dec_zone_state(struct zone *zone, enum zone_stat_item item) 265static inline void __dec_zone_state(struct zone *zone, enum zone_stat_item item)
223{ 266{
224 atomic_long_dec(&zone->vm_stat[item]); 267 atomic_long_dec(&zone->vm_stat[item]);
225 atomic_long_dec(&vm_stat[item]); 268 atomic_long_dec(&vm_zone_stat[item]);
269}
270
271static inline void __dec_node_state(struct pglist_data *pgdat, enum node_stat_item item)
272{
273 atomic_long_dec(&pgdat->vm_stat[item]);
274 atomic_long_dec(&vm_node_stat[item]);
226} 275}
227 276
228static inline void __inc_zone_page_state(struct page *page, 277static inline void __inc_zone_page_state(struct page *page,
@@ -231,12 +280,26 @@ static inline void __inc_zone_page_state(struct page *page,
231 __inc_zone_state(page_zone(page), item); 280 __inc_zone_state(page_zone(page), item);
232} 281}
233 282
283static inline void __inc_node_page_state(struct page *page,
284 enum node_stat_item item)
285{
286 __inc_node_state(page_pgdat(page), item);
287}
288
289
234static inline void __dec_zone_page_state(struct page *page, 290static inline void __dec_zone_page_state(struct page *page,
235 enum zone_stat_item item) 291 enum zone_stat_item item)
236{ 292{
237 __dec_zone_state(page_zone(page), item); 293 __dec_zone_state(page_zone(page), item);
238} 294}
239 295
296static inline void __dec_node_page_state(struct page *page,
297 enum node_stat_item item)
298{
299 __dec_node_state(page_pgdat(page), item);
300}
301
302
240/* 303/*
241 * We only use atomic operations to update counters. So there is no need to 304 * We only use atomic operations to update counters. So there is no need to
242 * disable interrupts. 305 * disable interrupts.
@@ -245,7 +308,12 @@ static inline void __dec_zone_page_state(struct page *page,
245#define dec_zone_page_state __dec_zone_page_state 308#define dec_zone_page_state __dec_zone_page_state
246#define mod_zone_page_state __mod_zone_page_state 309#define mod_zone_page_state __mod_zone_page_state
247 310
311#define inc_node_page_state __inc_node_page_state
312#define dec_node_page_state __dec_node_page_state
313#define mod_node_page_state __mod_node_page_state
314
248#define inc_zone_state __inc_zone_state 315#define inc_zone_state __inc_zone_state
316#define inc_node_state __inc_node_state
249#define dec_zone_state __dec_zone_state 317#define dec_zone_state __dec_zone_state
250 318
251#define set_pgdat_percpu_threshold(pgdat, callback) { } 319#define set_pgdat_percpu_threshold(pgdat, callback) { }
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index d7a7970eae21..7d4ff81b973f 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -4204,8 +4204,8 @@ void si_meminfo_node(struct sysinfo *val, int nid)
4204 for (zone_type = 0; zone_type < MAX_NR_ZONES; zone_type++) 4204 for (zone_type = 0; zone_type < MAX_NR_ZONES; zone_type++)
4205 managed_pages += pgdat->node_zones[zone_type].managed_pages; 4205 managed_pages += pgdat->node_zones[zone_type].managed_pages;
4206 val->totalram = managed_pages; 4206 val->totalram = managed_pages;
4207 val->sharedram = node_page_state(nid, NR_SHMEM); 4207 val->sharedram = sum_zone_node_page_state(nid, NR_SHMEM);
4208 val->freeram = node_page_state(nid, NR_FREE_PAGES); 4208 val->freeram = sum_zone_node_page_state(nid, NR_FREE_PAGES);
4209#ifdef CONFIG_HIGHMEM 4209#ifdef CONFIG_HIGHMEM
4210 for (zone_type = 0; zone_type < MAX_NR_ZONES; zone_type++) { 4210 for (zone_type = 0; zone_type < MAX_NR_ZONES; zone_type++) {
4211 struct zone *zone = &pgdat->node_zones[zone_type]; 4211 struct zone *zone = &pgdat->node_zones[zone_type];
@@ -5330,6 +5330,11 @@ static void __meminit setup_zone_pageset(struct zone *zone)
5330 zone->pageset = alloc_percpu(struct per_cpu_pageset); 5330 zone->pageset = alloc_percpu(struct per_cpu_pageset);
5331 for_each_possible_cpu(cpu) 5331 for_each_possible_cpu(cpu)
5332 zone_pageset_init(zone, cpu); 5332 zone_pageset_init(zone, cpu);
5333
5334 if (!zone->zone_pgdat->per_cpu_nodestats) {
5335 zone->zone_pgdat->per_cpu_nodestats =
5336 alloc_percpu(struct per_cpu_nodestat);
5337 }
5333} 5338}
5334 5339
5335/* 5340/*
@@ -6033,6 +6038,7 @@ void __paginginit free_area_init_node(int nid, unsigned long *zones_size,
6033 reset_deferred_meminit(pgdat); 6038 reset_deferred_meminit(pgdat);
6034 pgdat->node_id = nid; 6039 pgdat->node_id = nid;
6035 pgdat->node_start_pfn = node_start_pfn; 6040 pgdat->node_start_pfn = node_start_pfn;
6041 pgdat->per_cpu_nodestats = NULL;
6036#ifdef CONFIG_HAVE_MEMBLOCK_NODE_MAP 6042#ifdef CONFIG_HAVE_MEMBLOCK_NODE_MAP
6037 get_pfn_range_for_nid(nid, &start_pfn, &end_pfn); 6043 get_pfn_range_for_nid(nid, &start_pfn, &end_pfn);
6038 pr_info("Initmem setup node %d [mem %#018Lx-%#018Lx]\n", nid, 6044 pr_info("Initmem setup node %d [mem %#018Lx-%#018Lx]\n", nid,
diff --git a/mm/vmstat.c b/mm/vmstat.c
index 7997f52935c9..3345d396a99b 100644
--- a/mm/vmstat.c
+++ b/mm/vmstat.c
@@ -86,8 +86,10 @@ void vm_events_fold_cpu(int cpu)
86 * 86 *
87 * vm_stat contains the global counters 87 * vm_stat contains the global counters
88 */ 88 */
89atomic_long_t vm_stat[NR_VM_ZONE_STAT_ITEMS] __cacheline_aligned_in_smp; 89atomic_long_t vm_zone_stat[NR_VM_ZONE_STAT_ITEMS] __cacheline_aligned_in_smp;
90EXPORT_SYMBOL(vm_stat); 90atomic_long_t vm_node_stat[NR_VM_NODE_STAT_ITEMS] __cacheline_aligned_in_smp;
91EXPORT_SYMBOL(vm_zone_stat);
92EXPORT_SYMBOL(vm_node_stat);
91 93
92#ifdef CONFIG_SMP 94#ifdef CONFIG_SMP
93 95
@@ -167,19 +169,36 @@ int calculate_normal_threshold(struct zone *zone)
167 */ 169 */
168void refresh_zone_stat_thresholds(void) 170void refresh_zone_stat_thresholds(void)
169{ 171{
172 struct pglist_data *pgdat;
170 struct zone *zone; 173 struct zone *zone;
171 int cpu; 174 int cpu;
172 int threshold; 175 int threshold;
173 176
177 /* Zero current pgdat thresholds */
178 for_each_online_pgdat(pgdat) {
179 for_each_online_cpu(cpu) {
180 per_cpu_ptr(pgdat->per_cpu_nodestats, cpu)->stat_threshold = 0;
181 }
182 }
183
174 for_each_populated_zone(zone) { 184 for_each_populated_zone(zone) {
185 struct pglist_data *pgdat = zone->zone_pgdat;
175 unsigned long max_drift, tolerate_drift; 186 unsigned long max_drift, tolerate_drift;
176 187
177 threshold = calculate_normal_threshold(zone); 188 threshold = calculate_normal_threshold(zone);
178 189
179 for_each_online_cpu(cpu) 190 for_each_online_cpu(cpu) {
191 int pgdat_threshold;
192
180 per_cpu_ptr(zone->pageset, cpu)->stat_threshold 193 per_cpu_ptr(zone->pageset, cpu)->stat_threshold
181 = threshold; 194 = threshold;
182 195
196 /* Base nodestat threshold on the largest populated zone. */
197 pgdat_threshold = per_cpu_ptr(pgdat->per_cpu_nodestats, cpu)->stat_threshold;
198 per_cpu_ptr(pgdat->per_cpu_nodestats, cpu)->stat_threshold
199 = max(threshold, pgdat_threshold);
200 }
201
183 /* 202 /*
184 * Only set percpu_drift_mark if there is a danger that 203 * Only set percpu_drift_mark if there is a danger that
185 * NR_FREE_PAGES reports the low watermark is ok when in fact 204 * NR_FREE_PAGES reports the low watermark is ok when in fact
@@ -238,6 +257,26 @@ void __mod_zone_page_state(struct zone *zone, enum zone_stat_item item,
238} 257}
239EXPORT_SYMBOL(__mod_zone_page_state); 258EXPORT_SYMBOL(__mod_zone_page_state);
240 259
260void __mod_node_page_state(struct pglist_data *pgdat, enum node_stat_item item,
261 long delta)
262{
263 struct per_cpu_nodestat __percpu *pcp = pgdat->per_cpu_nodestats;
264 s8 __percpu *p = pcp->vm_node_stat_diff + item;
265 long x;
266 long t;
267
268 x = delta + __this_cpu_read(*p);
269
270 t = __this_cpu_read(pcp->stat_threshold);
271
272 if (unlikely(x > t || x < -t)) {
273 node_page_state_add(x, pgdat, item);
274 x = 0;
275 }
276 __this_cpu_write(*p, x);
277}
278EXPORT_SYMBOL(__mod_node_page_state);
279
241/* 280/*
242 * Optimized increment and decrement functions. 281 * Optimized increment and decrement functions.
243 * 282 *
@@ -277,12 +316,34 @@ void __inc_zone_state(struct zone *zone, enum zone_stat_item item)
277 } 316 }
278} 317}
279 318
319void __inc_node_state(struct pglist_data *pgdat, enum node_stat_item item)
320{
321 struct per_cpu_nodestat __percpu *pcp = pgdat->per_cpu_nodestats;
322 s8 __percpu *p = pcp->vm_node_stat_diff + item;
323 s8 v, t;
324
325 v = __this_cpu_inc_return(*p);
326 t = __this_cpu_read(pcp->stat_threshold);
327 if (unlikely(v > t)) {
328 s8 overstep = t >> 1;
329
330 node_page_state_add(v + overstep, pgdat, item);
331 __this_cpu_write(*p, -overstep);
332 }
333}
334
280void __inc_zone_page_state(struct page *page, enum zone_stat_item item) 335void __inc_zone_page_state(struct page *page, enum zone_stat_item item)
281{ 336{
282 __inc_zone_state(page_zone(page), item); 337 __inc_zone_state(page_zone(page), item);
283} 338}
284EXPORT_SYMBOL(__inc_zone_page_state); 339EXPORT_SYMBOL(__inc_zone_page_state);
285 340
341void __inc_node_page_state(struct page *page, enum node_stat_item item)
342{
343 __inc_node_state(page_pgdat(page), item);
344}
345EXPORT_SYMBOL(__inc_node_page_state);
346
286void __dec_zone_state(struct zone *zone, enum zone_stat_item item) 347void __dec_zone_state(struct zone *zone, enum zone_stat_item item)
287{ 348{
288 struct per_cpu_pageset __percpu *pcp = zone->pageset; 349 struct per_cpu_pageset __percpu *pcp = zone->pageset;
@@ -299,12 +360,34 @@ void __dec_zone_state(struct zone *zone, enum zone_stat_item item)
299 } 360 }
300} 361}
301 362
363void __dec_node_state(struct pglist_data *pgdat, enum node_stat_item item)
364{
365 struct per_cpu_nodestat __percpu *pcp = pgdat->per_cpu_nodestats;
366 s8 __percpu *p = pcp->vm_node_stat_diff + item;
367 s8 v, t;
368
369 v = __this_cpu_dec_return(*p);
370 t = __this_cpu_read(pcp->stat_threshold);
371 if (unlikely(v < - t)) {
372 s8 overstep = t >> 1;
373
374 node_page_state_add(v - overstep, pgdat, item);
375 __this_cpu_write(*p, overstep);
376 }
377}
378
302void __dec_zone_page_state(struct page *page, enum zone_stat_item item) 379void __dec_zone_page_state(struct page *page, enum zone_stat_item item)
303{ 380{
304 __dec_zone_state(page_zone(page), item); 381 __dec_zone_state(page_zone(page), item);
305} 382}
306EXPORT_SYMBOL(__dec_zone_page_state); 383EXPORT_SYMBOL(__dec_zone_page_state);
307 384
385void __dec_node_page_state(struct page *page, enum node_stat_item item)
386{
387 __dec_node_state(page_pgdat(page), item);
388}
389EXPORT_SYMBOL(__dec_node_page_state);
390
308#ifdef CONFIG_HAVE_CMPXCHG_LOCAL 391#ifdef CONFIG_HAVE_CMPXCHG_LOCAL
309/* 392/*
310 * If we have cmpxchg_local support then we do not need to incur the overhead 393 * If we have cmpxchg_local support then we do not need to incur the overhead
@@ -318,8 +401,8 @@ EXPORT_SYMBOL(__dec_zone_page_state);
318 * 1 Overstepping half of threshold 401 * 1 Overstepping half of threshold
319 * -1 Overstepping minus half of threshold 402 * -1 Overstepping minus half of threshold
320*/ 403*/
321static inline void mod_state(struct zone *zone, enum zone_stat_item item, 404static inline void mod_zone_state(struct zone *zone,
322 long delta, int overstep_mode) 405 enum zone_stat_item item, long delta, int overstep_mode)
323{ 406{
324 struct per_cpu_pageset __percpu *pcp = zone->pageset; 407 struct per_cpu_pageset __percpu *pcp = zone->pageset;
325 s8 __percpu *p = pcp->vm_stat_diff + item; 408 s8 __percpu *p = pcp->vm_stat_diff + item;
@@ -359,26 +442,88 @@ static inline void mod_state(struct zone *zone, enum zone_stat_item item,
359void mod_zone_page_state(struct zone *zone, enum zone_stat_item item, 442void mod_zone_page_state(struct zone *zone, enum zone_stat_item item,
360 long delta) 443 long delta)
361{ 444{
362 mod_state(zone, item, delta, 0); 445 mod_zone_state(zone, item, delta, 0);
363} 446}
364EXPORT_SYMBOL(mod_zone_page_state); 447EXPORT_SYMBOL(mod_zone_page_state);
365 448
366void inc_zone_state(struct zone *zone, enum zone_stat_item item) 449void inc_zone_state(struct zone *zone, enum zone_stat_item item)
367{ 450{
368 mod_state(zone, item, 1, 1); 451 mod_zone_state(zone, item, 1, 1);
369} 452}
370 453
371void inc_zone_page_state(struct page *page, enum zone_stat_item item) 454void inc_zone_page_state(struct page *page, enum zone_stat_item item)
372{ 455{
373 mod_state(page_zone(page), item, 1, 1); 456 mod_zone_state(page_zone(page), item, 1, 1);
374} 457}
375EXPORT_SYMBOL(inc_zone_page_state); 458EXPORT_SYMBOL(inc_zone_page_state);
376 459
377void dec_zone_page_state(struct page *page, enum zone_stat_item item) 460void dec_zone_page_state(struct page *page, enum zone_stat_item item)
378{ 461{
379 mod_state(page_zone(page), item, -1, -1); 462 mod_zone_state(page_zone(page), item, -1, -1);
380} 463}
381EXPORT_SYMBOL(dec_zone_page_state); 464EXPORT_SYMBOL(dec_zone_page_state);
465
466static inline void mod_node_state(struct pglist_data *pgdat,
467 enum node_stat_item item, int delta, int overstep_mode)
468{
469 struct per_cpu_nodestat __percpu *pcp = pgdat->per_cpu_nodestats;
470 s8 __percpu *p = pcp->vm_node_stat_diff + item;
471 long o, n, t, z;
472
473 do {
474 z = 0; /* overflow to node counters */
475
476 /*
477 * The fetching of the stat_threshold is racy. We may apply
478 * a counter threshold to the wrong the cpu if we get
479 * rescheduled while executing here. However, the next
480 * counter update will apply the threshold again and
481 * therefore bring the counter under the threshold again.
482 *
483 * Most of the time the thresholds are the same anyways
484 * for all cpus in a node.
485 */
486 t = this_cpu_read(pcp->stat_threshold);
487
488 o = this_cpu_read(*p);
489 n = delta + o;
490
491 if (n > t || n < -t) {
492 int os = overstep_mode * (t >> 1) ;
493
494 /* Overflow must be added to node counters */
495 z = n + os;
496 n = -os;
497 }
498 } while (this_cpu_cmpxchg(*p, o, n) != o);
499
500 if (z)
501 node_page_state_add(z, pgdat, item);
502}
503
504void mod_node_page_state(struct pglist_data *pgdat, enum node_stat_item item,
505 long delta)
506{
507 mod_node_state(pgdat, item, delta, 0);
508}
509EXPORT_SYMBOL(mod_node_page_state);
510
511void inc_node_state(struct pglist_data *pgdat, enum node_stat_item item)
512{
513 mod_node_state(pgdat, item, 1, 1);
514}
515
516void inc_node_page_state(struct page *page, enum node_stat_item item)
517{
518 mod_node_state(page_pgdat(page), item, 1, 1);
519}
520EXPORT_SYMBOL(inc_node_page_state);
521
522void dec_node_page_state(struct page *page, enum node_stat_item item)
523{
524 mod_node_state(page_pgdat(page), item, -1, -1);
525}
526EXPORT_SYMBOL(dec_node_page_state);
382#else 527#else
383/* 528/*
384 * Use interrupt disable to serialize counter updates 529 * Use interrupt disable to serialize counter updates
@@ -424,21 +569,69 @@ void dec_zone_page_state(struct page *page, enum zone_stat_item item)
424 local_irq_restore(flags); 569 local_irq_restore(flags);
425} 570}
426EXPORT_SYMBOL(dec_zone_page_state); 571EXPORT_SYMBOL(dec_zone_page_state);
427#endif
428 572
573void inc_node_state(struct pglist_data *pgdat, enum node_stat_item item)
574{
575 unsigned long flags;
576
577 local_irq_save(flags);
578 __inc_node_state(pgdat, item);
579 local_irq_restore(flags);
580}
581EXPORT_SYMBOL(inc_node_state);
582
583void mod_node_page_state(struct pglist_data *pgdat, enum node_stat_item item,
584 long delta)
585{
586 unsigned long flags;
587
588 local_irq_save(flags);
589 __mod_node_page_state(pgdat, item, delta);
590 local_irq_restore(flags);
591}
592EXPORT_SYMBOL(mod_node_page_state);
593
594void inc_node_page_state(struct page *page, enum node_stat_item item)
595{
596 unsigned long flags;
597 struct pglist_data *pgdat;
598
599 pgdat = page_pgdat(page);
600 local_irq_save(flags);
601 __inc_node_state(pgdat, item);
602 local_irq_restore(flags);
603}
604EXPORT_SYMBOL(inc_node_page_state);
605
606void dec_node_page_state(struct page *page, enum node_stat_item item)
607{
608 unsigned long flags;
609
610 local_irq_save(flags);
611 __dec_node_page_state(page, item);
612 local_irq_restore(flags);
613}
614EXPORT_SYMBOL(dec_node_page_state);
615#endif
429 616
430/* 617/*
431 * Fold a differential into the global counters. 618 * Fold a differential into the global counters.
432 * Returns the number of counters updated. 619 * Returns the number of counters updated.
433 */ 620 */
434static int fold_diff(int *diff) 621static int fold_diff(int *zone_diff, int *node_diff)
435{ 622{
436 int i; 623 int i;
437 int changes = 0; 624 int changes = 0;
438 625
439 for (i = 0; i < NR_VM_ZONE_STAT_ITEMS; i++) 626 for (i = 0; i < NR_VM_ZONE_STAT_ITEMS; i++)
440 if (diff[i]) { 627 if (zone_diff[i]) {
441 atomic_long_add(diff[i], &vm_stat[i]); 628 atomic_long_add(zone_diff[i], &vm_zone_stat[i]);
629 changes++;
630 }
631
632 for (i = 0; i < NR_VM_NODE_STAT_ITEMS; i++)
633 if (node_diff[i]) {
634 atomic_long_add(node_diff[i], &vm_node_stat[i]);
442 changes++; 635 changes++;
443 } 636 }
444 return changes; 637 return changes;
@@ -462,9 +655,11 @@ static int fold_diff(int *diff)
462 */ 655 */
463static int refresh_cpu_vm_stats(bool do_pagesets) 656static int refresh_cpu_vm_stats(bool do_pagesets)
464{ 657{
658 struct pglist_data *pgdat;
465 struct zone *zone; 659 struct zone *zone;
466 int i; 660 int i;
467 int global_diff[NR_VM_ZONE_STAT_ITEMS] = { 0, }; 661 int global_zone_diff[NR_VM_ZONE_STAT_ITEMS] = { 0, };
662 int global_node_diff[NR_VM_NODE_STAT_ITEMS] = { 0, };
468 int changes = 0; 663 int changes = 0;
469 664
470 for_each_populated_zone(zone) { 665 for_each_populated_zone(zone) {
@@ -477,7 +672,7 @@ static int refresh_cpu_vm_stats(bool do_pagesets)
477 if (v) { 672 if (v) {
478 673
479 atomic_long_add(v, &zone->vm_stat[i]); 674 atomic_long_add(v, &zone->vm_stat[i]);
480 global_diff[i] += v; 675 global_zone_diff[i] += v;
481#ifdef CONFIG_NUMA 676#ifdef CONFIG_NUMA
482 /* 3 seconds idle till flush */ 677 /* 3 seconds idle till flush */
483 __this_cpu_write(p->expire, 3); 678 __this_cpu_write(p->expire, 3);
@@ -516,7 +711,22 @@ static int refresh_cpu_vm_stats(bool do_pagesets)
516 } 711 }
517#endif 712#endif
518 } 713 }
519 changes += fold_diff(global_diff); 714
715 for_each_online_pgdat(pgdat) {
716 struct per_cpu_nodestat __percpu *p = pgdat->per_cpu_nodestats;
717
718 for (i = 0; i < NR_VM_NODE_STAT_ITEMS; i++) {
719 int v;
720
721 v = this_cpu_xchg(p->vm_node_stat_diff[i], 0);
722 if (v) {
723 atomic_long_add(v, &pgdat->vm_stat[i]);
724 global_node_diff[i] += v;
725 }
726 }
727 }
728
729 changes += fold_diff(global_zone_diff, global_node_diff);
520 return changes; 730 return changes;
521} 731}
522 732
@@ -527,9 +737,11 @@ static int refresh_cpu_vm_stats(bool do_pagesets)
527 */ 737 */
528void cpu_vm_stats_fold(int cpu) 738void cpu_vm_stats_fold(int cpu)
529{ 739{
740 struct pglist_data *pgdat;
530 struct zone *zone; 741 struct zone *zone;
531 int i; 742 int i;
532 int global_diff[NR_VM_ZONE_STAT_ITEMS] = { 0, }; 743 int global_zone_diff[NR_VM_ZONE_STAT_ITEMS] = { 0, };
744 int global_node_diff[NR_VM_NODE_STAT_ITEMS] = { 0, };
533 745
534 for_each_populated_zone(zone) { 746 for_each_populated_zone(zone) {
535 struct per_cpu_pageset *p; 747 struct per_cpu_pageset *p;
@@ -543,11 +755,27 @@ void cpu_vm_stats_fold(int cpu)
543 v = p->vm_stat_diff[i]; 755 v = p->vm_stat_diff[i];
544 p->vm_stat_diff[i] = 0; 756 p->vm_stat_diff[i] = 0;
545 atomic_long_add(v, &zone->vm_stat[i]); 757 atomic_long_add(v, &zone->vm_stat[i]);
546 global_diff[i] += v; 758 global_zone_diff[i] += v;
547 } 759 }
548 } 760 }
549 761
550 fold_diff(global_diff); 762 for_each_online_pgdat(pgdat) {
763 struct per_cpu_nodestat *p;
764
765 p = per_cpu_ptr(pgdat->per_cpu_nodestats, cpu);
766
767 for (i = 0; i < NR_VM_NODE_STAT_ITEMS; i++)
768 if (p->vm_node_stat_diff[i]) {
769 int v;
770
771 v = p->vm_node_stat_diff[i];
772 p->vm_node_stat_diff[i] = 0;
773 atomic_long_add(v, &pgdat->vm_stat[i]);
774 global_node_diff[i] += v;
775 }
776 }
777
778 fold_diff(global_zone_diff, global_node_diff);
551} 779}
552 780
553/* 781/*
@@ -563,16 +791,19 @@ void drain_zonestat(struct zone *zone, struct per_cpu_pageset *pset)
563 int v = pset->vm_stat_diff[i]; 791 int v = pset->vm_stat_diff[i];
564 pset->vm_stat_diff[i] = 0; 792 pset->vm_stat_diff[i] = 0;
565 atomic_long_add(v, &zone->vm_stat[i]); 793 atomic_long_add(v, &zone->vm_stat[i]);
566 atomic_long_add(v, &vm_stat[i]); 794 atomic_long_add(v, &vm_zone_stat[i]);
567 } 795 }
568} 796}
569#endif 797#endif
570 798
571#ifdef CONFIG_NUMA 799#ifdef CONFIG_NUMA
572/* 800/*
573 * Determine the per node value of a stat item. 801 * Determine the per node value of a stat item. This function
802 * is called frequently in a NUMA machine, so try to be as
803 * frugal as possible.
574 */ 804 */
575unsigned long node_page_state(int node, enum zone_stat_item item) 805unsigned long sum_zone_node_page_state(int node,
806 enum zone_stat_item item)
576{ 807{
577 struct zone *zones = NODE_DATA(node)->node_zones; 808 struct zone *zones = NODE_DATA(node)->node_zones;
578 int i; 809 int i;
@@ -584,6 +815,19 @@ unsigned long node_page_state(int node, enum zone_stat_item item)
584 return count; 815 return count;
585} 816}
586 817
818/*
819 * Determine the per node value of a stat item.
820 */
821unsigned long node_page_state(struct pglist_data *pgdat,
822 enum node_stat_item item)
823{
824 long x = atomic_long_read(&pgdat->vm_stat[item]);
825#ifdef CONFIG_SMP
826 if (x < 0)
827 x = 0;
828#endif
829 return x;
830}
587#endif 831#endif
588 832
589#ifdef CONFIG_COMPACTION 833#ifdef CONFIG_COMPACTION
@@ -1287,6 +1531,7 @@ static void *vmstat_start(struct seq_file *m, loff_t *pos)
1287 if (*pos >= ARRAY_SIZE(vmstat_text)) 1531 if (*pos >= ARRAY_SIZE(vmstat_text))
1288 return NULL; 1532 return NULL;
1289 stat_items_size = NR_VM_ZONE_STAT_ITEMS * sizeof(unsigned long) + 1533 stat_items_size = NR_VM_ZONE_STAT_ITEMS * sizeof(unsigned long) +
1534 NR_VM_NODE_STAT_ITEMS * sizeof(unsigned long) +
1290 NR_VM_WRITEBACK_STAT_ITEMS * sizeof(unsigned long); 1535 NR_VM_WRITEBACK_STAT_ITEMS * sizeof(unsigned long);
1291 1536
1292#ifdef CONFIG_VM_EVENT_COUNTERS 1537#ifdef CONFIG_VM_EVENT_COUNTERS
@@ -1301,6 +1546,10 @@ static void *vmstat_start(struct seq_file *m, loff_t *pos)
1301 v[i] = global_page_state(i); 1546 v[i] = global_page_state(i);
1302 v += NR_VM_ZONE_STAT_ITEMS; 1547 v += NR_VM_ZONE_STAT_ITEMS;
1303 1548
1549 for (i = 0; i < NR_VM_NODE_STAT_ITEMS; i++)
1550 v[i] = global_node_page_state(i);
1551 v += NR_VM_NODE_STAT_ITEMS;
1552
1304 global_dirty_limits(v + NR_DIRTY_BG_THRESHOLD, 1553 global_dirty_limits(v + NR_DIRTY_BG_THRESHOLD,
1305 v + NR_DIRTY_THRESHOLD); 1554 v + NR_DIRTY_THRESHOLD);
1306 v += NR_VM_WRITEBACK_STAT_ITEMS; 1555 v += NR_VM_WRITEBACK_STAT_ITEMS;
@@ -1390,7 +1639,7 @@ int vmstat_refresh(struct ctl_table *table, int write,
1390 if (err) 1639 if (err)
1391 return err; 1640 return err;
1392 for (i = 0; i < NR_VM_ZONE_STAT_ITEMS; i++) { 1641 for (i = 0; i < NR_VM_ZONE_STAT_ITEMS; i++) {
1393 val = atomic_long_read(&vm_stat[i]); 1642 val = atomic_long_read(&vm_zone_stat[i]);
1394 if (val < 0) { 1643 if (val < 0) {
1395 switch (i) { 1644 switch (i) {
1396 case NR_ALLOC_BATCH: 1645 case NR_ALLOC_BATCH:
diff --git a/mm/workingset.c b/mm/workingset.c
index d7cc4bbd7e1b..5ffba0c0adc6 100644
--- a/mm/workingset.c
+++ b/mm/workingset.c
@@ -351,12 +351,13 @@ static unsigned long count_shadow_nodes(struct shrinker *shrinker,
351 shadow_nodes = list_lru_shrink_count(&workingset_shadow_nodes, sc); 351 shadow_nodes = list_lru_shrink_count(&workingset_shadow_nodes, sc);
352 local_irq_enable(); 352 local_irq_enable();
353 353
354 if (memcg_kmem_enabled()) 354 if (memcg_kmem_enabled()) {
355 pages = mem_cgroup_node_nr_lru_pages(sc->memcg, sc->nid, 355 pages = mem_cgroup_node_nr_lru_pages(sc->memcg, sc->nid,
356 LRU_ALL_FILE); 356 LRU_ALL_FILE);
357 else 357 } else {
358 pages = node_page_state(sc->nid, NR_ACTIVE_FILE) + 358 pages = sum_zone_node_page_state(sc->nid, NR_ACTIVE_FILE) +
359 node_page_state(sc->nid, NR_INACTIVE_FILE); 359 sum_zone_node_page_state(sc->nid, NR_INACTIVE_FILE);
360 }
360 361
361 /* 362 /*
362 * Active cache pages are limited to 50% of memory, and shadow 363 * Active cache pages are limited to 50% of memory, and shadow
generated by cgit v1.2.2 (git 2.25.0) at 2025-09-16 22:09:59 -0400