aboutsummaryrefslogtreecommitdiffstats
path: root/Documentation/cgroups/memory.txt
diff options
context:
space:
mode:
authorKAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>2010-05-26 17:42:40 -0400
committerLinus Torvalds <torvalds@linux-foundation.org>2010-05-27 12:12:43 -0400
commitdc10e281f5fc42e288ab979294d1d5dc9743ae1b (patch)
tree186ab100831ae92b10cd543eb666372f51917ab8 /Documentation/cgroups/memory.txt
parent87946a72283be3de936adc754b7007df7d3e6aeb (diff)
memcg: update documentation
Some information are old, and I think current document doesn't work as "a guide for users". We need summary of all of our controls, at least. Signed-off-by: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com> Reviewed-by: Randy Dunlap <randy.dunlap@oracle.com> Cc: Balbir Singh <balbir@in.ibm.com> Cc: Daisuke Nishimura <nishimura@mxp.nes.nec.co.jp> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Diffstat (limited to 'Documentation/cgroups/memory.txt')
-rw-r--r--Documentation/cgroups/memory.txt291
1 files changed, 198 insertions, 93 deletions
diff --git a/Documentation/cgroups/memory.txt b/Documentation/cgroups/memory.txt
index 5e028870ee8a..7781857dc940 100644
--- a/Documentation/cgroups/memory.txt
+++ b/Documentation/cgroups/memory.txt
@@ -1,18 +1,15 @@
1Memory Resource Controller 1Memory Resource Controller
2 2
3NOTE: The Memory Resource Controller has been generically been referred 3NOTE: The Memory Resource Controller has been generically been referred
4to as the memory controller in this document. Do not confuse memory controller 4 to as the memory controller in this document. Do not confuse memory
5used here with the memory controller that is used in hardware. 5 controller used here with the memory controller that is used in hardware.
6 6
7Salient features 7(For editors)
8 8In this document:
9a. Enable control of Anonymous, Page Cache (mapped and unmapped) and 9 When we mention a cgroup (cgroupfs's directory) with memory controller,
10 Swap Cache memory pages. 10 we call it "memory cgroup". When you see git-log and source code, you'll
11b. The infrastructure allows easy addition of other types of memory to control 11 see patch's title and function names tend to use "memcg".
12c. Provides *zero overhead* for non memory controller users 12 In this document, we avoid using it.
13d. Provides a double LRU: global memory pressure causes reclaim from the
14 global LRU; a cgroup on hitting a limit, reclaims from the per
15 cgroup LRU
16 13
17Benefits and Purpose of the memory controller 14Benefits and Purpose of the memory controller
18 15
@@ -33,6 +30,45 @@ d. A CD/DVD burner could control the amount of memory used by the
33e. There are several other use cases, find one or use the controller just 30e. There are several other use cases, find one or use the controller just
34 for fun (to learn and hack on the VM subsystem). 31 for fun (to learn and hack on the VM subsystem).
35 32
33Current Status: linux-2.6.34-mmotm(development version of 2010/April)
34
35Features:
36 - accounting anonymous pages, file caches, swap caches usage and limiting them.
37 - private LRU and reclaim routine. (system's global LRU and private LRU
38 work independently from each other)
39 - optionally, memory+swap usage can be accounted and limited.
40 - hierarchical accounting
41 - soft limit
42 - moving(recharging) account at moving a task is selectable.
43 - usage threshold notifier
44 - oom-killer disable knob and oom-notifier
45 - Root cgroup has no limit controls.
46
47 Kernel memory and Hugepages are not under control yet. We just manage
48 pages on LRU. To add more controls, we have to take care of performance.
49
50Brief summary of control files.
51
52 tasks # attach a task(thread) and show list of threads
53 cgroup.procs # show list of processes
54 cgroup.event_control # an interface for event_fd()
55 memory.usage_in_bytes # show current memory(RSS+Cache) usage.
56 memory.memsw.usage_in_bytes # show current memory+Swap usage
57 memory.limit_in_bytes # set/show limit of memory usage
58 memory.memsw.limit_in_bytes # set/show limit of memory+Swap usage
59 memory.failcnt # show the number of memory usage hits limits
60 memory.memsw.failcnt # show the number of memory+Swap hits limits
61 memory.max_usage_in_bytes # show max memory usage recorded
62 memory.memsw.usage_in_bytes # show max memory+Swap usage recorded
63 memory.soft_limit_in_bytes # set/show soft limit of memory usage
64 memory.stat # show various statistics
65 memory.use_hierarchy # set/show hierarchical account enabled
66 memory.force_empty # trigger forced move charge to parent
67 memory.swappiness # set/show swappiness parameter of vmscan
68 (See sysctl's vm.swappiness)
69 memory.move_charge_at_immigrate # set/show controls of moving charges
70 memory.oom_control # set/show oom controls.
71
361. History 721. History
37 73
38The memory controller has a long history. A request for comments for the memory 74The memory controller has a long history. A request for comments for the memory
@@ -106,14 +142,14 @@ the necessary data structures and check if the cgroup that is being charged
106is over its limit. If it is then reclaim is invoked on the cgroup. 142is over its limit. If it is then reclaim is invoked on the cgroup.
107More details can be found in the reclaim section of this document. 143More details can be found in the reclaim section of this document.
108If everything goes well, a page meta-data-structure called page_cgroup is 144If everything goes well, a page meta-data-structure called page_cgroup is
109allocated and associated with the page. This routine also adds the page to 145updated. page_cgroup has its own LRU on cgroup.
110the per cgroup LRU. 146(*) page_cgroup structure is allocated at boot/memory-hotplug time.
111 147
1122.2.1 Accounting details 1482.2.1 Accounting details
113 149
114All mapped anon pages (RSS) and cache pages (Page Cache) are accounted. 150All mapped anon pages (RSS) and cache pages (Page Cache) are accounted.
115(some pages which never be reclaimable and will not be on global LRU 151Some pages which are never reclaimable and will not be on the global LRU
116 are not accounted. we just accounts pages under usual vm management.) 152are not accounted. We just account pages under usual VM management.
117 153
118RSS pages are accounted at page_fault unless they've already been accounted 154RSS pages are accounted at page_fault unless they've already been accounted
119for earlier. A file page will be accounted for as Page Cache when it's 155for earlier. A file page will be accounted for as Page Cache when it's
@@ -121,12 +157,19 @@ inserted into inode (radix-tree). While it's mapped into the page tables of
121processes, duplicate accounting is carefully avoided. 157processes, duplicate accounting is carefully avoided.
122 158
123A RSS page is unaccounted when it's fully unmapped. A PageCache page is 159A RSS page is unaccounted when it's fully unmapped. A PageCache page is
124unaccounted when it's removed from radix-tree. 160unaccounted when it's removed from radix-tree. Even if RSS pages are fully
161unmapped (by kswapd), they may exist as SwapCache in the system until they
162are really freed. Such SwapCaches also also accounted.
163A swapped-in page is not accounted until it's mapped.
164
165Note: The kernel does swapin-readahead and read multiple swaps at once.
166This means swapped-in pages may contain pages for other tasks than a task
167causing page fault. So, we avoid accounting at swap-in I/O.
125 168
126At page migration, accounting information is kept. 169At page migration, accounting information is kept.
127 170
128Note: we just account pages-on-lru because our purpose is to control amount 171Note: we just account pages-on-LRU because our purpose is to control amount
129of used pages. not-on-lru pages are tend to be out-of-control from vm view. 172of used pages; not-on-LRU pages tend to be out-of-control from VM view.
130 173
1312.3 Shared Page Accounting 1742.3 Shared Page Accounting
132 175
@@ -143,6 +186,7 @@ caller of swapoff rather than the users of shmem.
143 186
144 187
1452.4 Swap Extension (CONFIG_CGROUP_MEM_RES_CTLR_SWAP) 1882.4 Swap Extension (CONFIG_CGROUP_MEM_RES_CTLR_SWAP)
189
146Swap Extension allows you to record charge for swap. A swapped-in page is 190Swap Extension allows you to record charge for swap. A swapped-in page is
147charged back to original page allocator if possible. 191charged back to original page allocator if possible.
148 192
@@ -150,13 +194,20 @@ When swap is accounted, following files are added.
150 - memory.memsw.usage_in_bytes. 194 - memory.memsw.usage_in_bytes.
151 - memory.memsw.limit_in_bytes. 195 - memory.memsw.limit_in_bytes.
152 196
153usage of mem+swap is limited by memsw.limit_in_bytes. 197memsw means memory+swap. Usage of memory+swap is limited by
198memsw.limit_in_bytes.
199
200Example: Assume a system with 4G of swap. A task which allocates 6G of memory
201(by mistake) under 2G memory limitation will use all swap.
202In this case, setting memsw.limit_in_bytes=3G will prevent bad use of swap.
203By using memsw limit, you can avoid system OOM which can be caused by swap
204shortage.
154 205
155* why 'mem+swap' rather than swap. 206* why 'memory+swap' rather than swap.
156The global LRU(kswapd) can swap out arbitrary pages. Swap-out means 207The global LRU(kswapd) can swap out arbitrary pages. Swap-out means
157to move account from memory to swap...there is no change in usage of 208to move account from memory to swap...there is no change in usage of
158mem+swap. In other words, when we want to limit the usage of swap without 209memory+swap. In other words, when we want to limit the usage of swap without
159affecting global LRU, mem+swap limit is better than just limiting swap from 210affecting global LRU, memory+swap limit is better than just limiting swap from
160OS point of view. 211OS point of view.
161 212
162* What happens when a cgroup hits memory.memsw.limit_in_bytes 213* What happens when a cgroup hits memory.memsw.limit_in_bytes
@@ -168,12 +219,12 @@ it by cgroup.
168 219
1692.5 Reclaim 2202.5 Reclaim
170 221
171Each cgroup maintains a per cgroup LRU that consists of an active 222Each cgroup maintains a per cgroup LRU which has the same structure as
172and inactive list. When a cgroup goes over its limit, we first try 223global VM. When a cgroup goes over its limit, we first try
173to reclaim memory from the cgroup so as to make space for the new 224to reclaim memory from the cgroup so as to make space for the new
174pages that the cgroup has touched. If the reclaim is unsuccessful, 225pages that the cgroup has touched. If the reclaim is unsuccessful,
175an OOM routine is invoked to select and kill the bulkiest task in the 226an OOM routine is invoked to select and kill the bulkiest task in the
176cgroup. 227cgroup. (See 10. OOM Control below.)
177 228
178The reclaim algorithm has not been modified for cgroups, except that 229The reclaim algorithm has not been modified for cgroups, except that
179pages that are selected for reclaiming come from the per cgroup LRU 230pages that are selected for reclaiming come from the per cgroup LRU
@@ -187,13 +238,19 @@ Note2: When panic_on_oom is set to "2", the whole system will panic.
187When oom event notifier is registered, event will be delivered. 238When oom event notifier is registered, event will be delivered.
188(See oom_control section) 239(See oom_control section)
189 240
1902. Locking 2412.6 Locking
191 242
192The memory controller uses the following hierarchy 243 lock_page_cgroup()/unlock_page_cgroup() should not be called under
244 mapping->tree_lock.
193 245
1941. zone->lru_lock is used for selecting pages to be isolated 246 Other lock order is following:
1952. mem->per_zone->lru_lock protects the per cgroup LRU (per zone) 247 PG_locked.
1963. lock_page_cgroup() is used to protect page->page_cgroup 248 mm->page_table_lock
249 zone->lru_lock
250 lock_page_cgroup.
251 In many cases, just lock_page_cgroup() is called.
252 per-zone-per-cgroup LRU (cgroup's private LRU) is just guarded by
253 zone->lru_lock, it has no lock of its own.
197 254
1983. User Interface 2553. User Interface
199 256
@@ -202,6 +259,7 @@ The memory controller uses the following hierarchy
202a. Enable CONFIG_CGROUPS 259a. Enable CONFIG_CGROUPS
203b. Enable CONFIG_RESOURCE_COUNTERS 260b. Enable CONFIG_RESOURCE_COUNTERS
204c. Enable CONFIG_CGROUP_MEM_RES_CTLR 261c. Enable CONFIG_CGROUP_MEM_RES_CTLR
262d. Enable CONFIG_CGROUP_MEM_RES_CTLR_SWAP (to use swap extension)
205 263
2061. Prepare the cgroups 2641. Prepare the cgroups
207# mkdir -p /cgroups 265# mkdir -p /cgroups
@@ -209,31 +267,28 @@ c. Enable CONFIG_CGROUP_MEM_RES_CTLR
209 267
2102. Make the new group and move bash into it 2682. Make the new group and move bash into it
211# mkdir /cgroups/0 269# mkdir /cgroups/0
212# echo $$ > /cgroups/0/tasks 270# echo $$ > /cgroups/0/tasks
213 271
214Since now we're in the 0 cgroup, 272Since now we're in the 0 cgroup, we can alter the memory limit:
215We can alter the memory limit:
216# echo 4M > /cgroups/0/memory.limit_in_bytes 273# echo 4M > /cgroups/0/memory.limit_in_bytes
217 274
218NOTE: We can use a suffix (k, K, m, M, g or G) to indicate values in kilo, 275NOTE: We can use a suffix (k, K, m, M, g or G) to indicate values in kilo,
219mega or gigabytes. 276mega or gigabytes. (Here, Kilo, Mega, Giga are Kibibytes, Mebibytes, Gibibytes.)
277
220NOTE: We can write "-1" to reset the *.limit_in_bytes(unlimited). 278NOTE: We can write "-1" to reset the *.limit_in_bytes(unlimited).
221NOTE: We cannot set limits on the root cgroup any more. 279NOTE: We cannot set limits on the root cgroup any more.
222 280
223# cat /cgroups/0/memory.limit_in_bytes 281# cat /cgroups/0/memory.limit_in_bytes
2244194304 2824194304
225 283
226NOTE: The interface has now changed to display the usage in bytes
227instead of pages
228
229We can check the usage: 284We can check the usage:
230# cat /cgroups/0/memory.usage_in_bytes 285# cat /cgroups/0/memory.usage_in_bytes
2311216512 2861216512
232 287
233A successful write to this file does not guarantee a successful set of 288A successful write to this file does not guarantee a successful set of
234this limit to the value written into the file. This can be due to a 289this limit to the value written into the file. This can be due to a
235number of factors, such as rounding up to page boundaries or the total 290number of factors, such as rounding up to page boundaries or the total
236availability of memory on the system. The user is required to re-read 291availability of memory on the system. The user is required to re-read
237this file after a write to guarantee the value committed by the kernel. 292this file after a write to guarantee the value committed by the kernel.
238 293
239# echo 1 > memory.limit_in_bytes 294# echo 1 > memory.limit_in_bytes
@@ -248,15 +303,23 @@ caches, RSS and Active pages/Inactive pages are shown.
248 303
2494. Testing 3044. Testing
250 305
251Balbir posted lmbench, AIM9, LTP and vmmstress results [10] and [11]. 306For testing features and implementation, see memcg_test.txt.
252Apart from that v6 has been tested with several applications and regular 307
253daily use. The controller has also been tested on the PPC64, x86_64 and 308Performance test is also important. To see pure memory controller's overhead,
254UML platforms. 309testing on tmpfs will give you good numbers of small overheads.
310Example: do kernel make on tmpfs.
311
312Page-fault scalability is also important. At measuring parallel
313page fault test, multi-process test may be better than multi-thread
314test because it has noise of shared objects/status.
315
316But the above two are testing extreme situations.
317Trying usual test under memory controller is always helpful.
255 318
2564.1 Troubleshooting 3194.1 Troubleshooting
257 320
258Sometimes a user might find that the application under a cgroup is 321Sometimes a user might find that the application under a cgroup is
259terminated. There are several causes for this: 322terminated by OOM killer. There are several causes for this:
260 323
2611. The cgroup limit is too low (just too low to do anything useful) 3241. The cgroup limit is too low (just too low to do anything useful)
2622. The user is using anonymous memory and swap is turned off or too low 3252. The user is using anonymous memory and swap is turned off or too low
@@ -264,6 +327,9 @@ terminated. There are several causes for this:
264A sync followed by echo 1 > /proc/sys/vm/drop_caches will help get rid of 327A sync followed by echo 1 > /proc/sys/vm/drop_caches will help get rid of
265some of the pages cached in the cgroup (page cache pages). 328some of the pages cached in the cgroup (page cache pages).
266 329
330To know what happens, disable OOM_Kill by 10. OOM Control(see below) and
331seeing what happens will be helpful.
332
2674.2 Task migration 3334.2 Task migration
268 334
269When a task migrates from one cgroup to another, its charge is not 335When a task migrates from one cgroup to another, its charge is not
@@ -271,16 +337,19 @@ carried forward by default. The pages allocated from the original cgroup still
271remain charged to it, the charge is dropped when the page is freed or 337remain charged to it, the charge is dropped when the page is freed or
272reclaimed. 338reclaimed.
273 339
274Note: You can move charges of a task along with task migration. See 8. 340You can move charges of a task along with task migration.
341See 8. "Move charges at task migration"
275 342
2764.3 Removing a cgroup 3434.3 Removing a cgroup
277 344
278A cgroup can be removed by rmdir, but as discussed in sections 4.1 and 4.2, a 345A cgroup can be removed by rmdir, but as discussed in sections 4.1 and 4.2, a
279cgroup might have some charge associated with it, even though all 346cgroup might have some charge associated with it, even though all
280tasks have migrated away from it. 347tasks have migrated away from it. (because we charge against pages, not
281Such charges are freed(at default) or moved to its parent. When moved, 348against tasks.)
282both of RSS and CACHES are moved to parent. 349
283If both of them are busy, rmdir() returns -EBUSY. See 5.1 Also. 350Such charges are freed or moved to their parent. At moving, both of RSS
351and CACHES are moved to parent.
352rmdir() may return -EBUSY if freeing/moving fails. See 5.1 also.
284 353
285Charges recorded in swap information is not updated at removal of cgroup. 354Charges recorded in swap information is not updated at removal of cgroup.
286Recorded information is discarded and a cgroup which uses swap (swapcache) 355Recorded information is discarded and a cgroup which uses swap (swapcache)
@@ -296,10 +365,10 @@ will be charged as a new owner of it.
296 365
297 # echo 0 > memory.force_empty 366 # echo 0 > memory.force_empty
298 367
299 Almost all pages tracked by this memcg will be unmapped and freed. Some of 368 Almost all pages tracked by this memory cgroup will be unmapped and freed.
300 pages cannot be freed because it's locked or in-use. Such pages are moved 369 Some pages cannot be freed because they are locked or in-use. Such pages are
301 to parent and this cgroup will be empty. But this may return -EBUSY in 370 moved to parent and this cgroup will be empty. This may return -EBUSY if
302 some too busy case. 371 VM is too busy to free/move all pages immediately.
303 372
304 Typical use case of this interface is that calling this before rmdir(). 373 Typical use case of this interface is that calling this before rmdir().
305 Because rmdir() moves all pages to parent, some out-of-use page caches can be 374 Because rmdir() moves all pages to parent, some out-of-use page caches can be
@@ -309,19 +378,41 @@ will be charged as a new owner of it.
309 378
310memory.stat file includes following statistics 379memory.stat file includes following statistics
311 380
381# per-memory cgroup local status
312cache - # of bytes of page cache memory. 382cache - # of bytes of page cache memory.
313rss - # of bytes of anonymous and swap cache memory. 383rss - # of bytes of anonymous and swap cache memory.
384mapped_file - # of bytes of mapped file (includes tmpfs/shmem)
314pgpgin - # of pages paged in (equivalent to # of charging events). 385pgpgin - # of pages paged in (equivalent to # of charging events).
315pgpgout - # of pages paged out (equivalent to # of uncharging events). 386pgpgout - # of pages paged out (equivalent to # of uncharging events).
316active_anon - # of bytes of anonymous and swap cache memory on active 387swap - # of bytes of swap usage
317 lru list.
318inactive_anon - # of bytes of anonymous memory and swap cache memory on 388inactive_anon - # of bytes of anonymous memory and swap cache memory on
319 inactive lru list. 389 LRU list.
320active_file - # of bytes of file-backed memory on active lru list. 390active_anon - # of bytes of anonymous and swap cache memory on active
321inactive_file - # of bytes of file-backed memory on inactive lru list. 391 inactive LRU list.
392inactive_file - # of bytes of file-backed memory on inactive LRU list.
393active_file - # of bytes of file-backed memory on active LRU list.
322unevictable - # of bytes of memory that cannot be reclaimed (mlocked etc). 394unevictable - # of bytes of memory that cannot be reclaimed (mlocked etc).
323 395
324The following additional stats are dependent on CONFIG_DEBUG_VM. 396# status considering hierarchy (see memory.use_hierarchy settings)
397
398hierarchical_memory_limit - # of bytes of memory limit with regard to hierarchy
399 under which the memory cgroup is
400hierarchical_memsw_limit - # of bytes of memory+swap limit with regard to
401 hierarchy under which memory cgroup is.
402
403total_cache - sum of all children's "cache"
404total_rss - sum of all children's "rss"
405total_mapped_file - sum of all children's "cache"
406total_pgpgin - sum of all children's "pgpgin"
407total_pgpgout - sum of all children's "pgpgout"
408total_swap - sum of all children's "swap"
409total_inactive_anon - sum of all children's "inactive_anon"
410total_active_anon - sum of all children's "active_anon"
411total_inactive_file - sum of all children's "inactive_file"
412total_active_file - sum of all children's "active_file"
413total_unevictable - sum of all children's "unevictable"
414
415# The following additional stats are dependent on CONFIG_DEBUG_VM.
325 416
326inactive_ratio - VM internal parameter. (see mm/page_alloc.c) 417inactive_ratio - VM internal parameter. (see mm/page_alloc.c)
327recent_rotated_anon - VM internal parameter. (see mm/vmscan.c) 418recent_rotated_anon - VM internal parameter. (see mm/vmscan.c)
@@ -330,24 +421,37 @@ recent_scanned_anon - VM internal parameter. (see mm/vmscan.c)
330recent_scanned_file - VM internal parameter. (see mm/vmscan.c) 421recent_scanned_file - VM internal parameter. (see mm/vmscan.c)
331 422
332Memo: 423Memo:
333 recent_rotated means recent frequency of lru rotation. 424 recent_rotated means recent frequency of LRU rotation.
334 recent_scanned means recent # of scans to lru. 425 recent_scanned means recent # of scans to LRU.
335 showing for better debug please see the code for meanings. 426 showing for better debug please see the code for meanings.
336 427
337Note: 428Note:
338 Only anonymous and swap cache memory is listed as part of 'rss' stat. 429 Only anonymous and swap cache memory is listed as part of 'rss' stat.
339 This should not be confused with the true 'resident set size' or the 430 This should not be confused with the true 'resident set size' or the
340 amount of physical memory used by the cgroup. Per-cgroup rss 431 amount of physical memory used by the cgroup.
341 accounting is not done yet. 432 'rss + file_mapped" will give you resident set size of cgroup.
433 (Note: file and shmem may be shared among other cgroups. In that case,
434 file_mapped is accounted only when the memory cgroup is owner of page
435 cache.)
342 436
3435.3 swappiness 4375.3 swappiness
344 Similar to /proc/sys/vm/swappiness, but affecting a hierarchy of groups only.
345 438
346 Following cgroups' swappiness can't be changed. 439Similar to /proc/sys/vm/swappiness, but affecting a hierarchy of groups only.
347 - root cgroup (uses /proc/sys/vm/swappiness).
348 - a cgroup which uses hierarchy and it has child cgroup.
349 - a cgroup which uses hierarchy and not the root of hierarchy.
350 440
441Following cgroups' swappiness can't be changed.
442- root cgroup (uses /proc/sys/vm/swappiness).
443- a cgroup which uses hierarchy and it has other cgroup(s) below it.
444- a cgroup which uses hierarchy and not the root of hierarchy.
445
4465.4 failcnt
447
448A memory cgroup provides memory.failcnt and memory.memsw.failcnt files.
449This failcnt(== failure count) shows the number of times that a usage counter
450hit its limit. When a memory cgroup hits a limit, failcnt increases and
451memory under it will be reclaimed.
452
453You can reset failcnt by writing 0 to failcnt file.
454# echo 0 > .../memory.failcnt
351 455
3526. Hierarchy support 4566. Hierarchy support
353 457
@@ -366,13 +470,13 @@ hierarchy
366 470
367In the diagram above, with hierarchical accounting enabled, all memory 471In the diagram above, with hierarchical accounting enabled, all memory
368usage of e, is accounted to its ancestors up until the root (i.e, c and root), 472usage of e, is accounted to its ancestors up until the root (i.e, c and root),
369that has memory.use_hierarchy enabled. If one of the ancestors goes over its 473that has memory.use_hierarchy enabled. If one of the ancestors goes over its
370limit, the reclaim algorithm reclaims from the tasks in the ancestor and the 474limit, the reclaim algorithm reclaims from the tasks in the ancestor and the
371children of the ancestor. 475children of the ancestor.
372 476
3736.1 Enabling hierarchical accounting and reclaim 4776.1 Enabling hierarchical accounting and reclaim
374 478
375The memory controller by default disables the hierarchy feature. Support 479A memory cgroup by default disables the hierarchy feature. Support
376can be enabled by writing 1 to memory.use_hierarchy file of the root cgroup 480can be enabled by writing 1 to memory.use_hierarchy file of the root cgroup
377 481
378# echo 1 > memory.use_hierarchy 482# echo 1 > memory.use_hierarchy
@@ -382,10 +486,10 @@ The feature can be disabled by
382# echo 0 > memory.use_hierarchy 486# echo 0 > memory.use_hierarchy
383 487
384NOTE1: Enabling/disabling will fail if the cgroup already has other 488NOTE1: Enabling/disabling will fail if the cgroup already has other
385cgroups created below it. 489 cgroups created below it.
386 490
387NOTE2: When panic_on_oom is set to "2", the whole system will panic in 491NOTE2: When panic_on_oom is set to "2", the whole system will panic in
388case of an oom event in any cgroup. 492 case of an OOM event in any cgroup.
389 493
3907. Soft limits 4947. Soft limits
391 495
@@ -395,7 +499,7 @@ is to allow control groups to use as much of the memory as needed, provided
395a. There is no memory contention 499a. There is no memory contention
396b. They do not exceed their hard limit 500b. They do not exceed their hard limit
397 501
398When the system detects memory contention or low memory control groups 502When the system detects memory contention or low memory, control groups
399are pushed back to their soft limits. If the soft limit of each control 503are pushed back to their soft limits. If the soft limit of each control
400group is very high, they are pushed back as much as possible to make 504group is very high, they are pushed back as much as possible to make
401sure that one control group does not starve the others of memory. 505sure that one control group does not starve the others of memory.
@@ -409,7 +513,7 @@ it gets invoked from balance_pgdat (kswapd).
4097.1 Interface 5137.1 Interface
410 514
411Soft limits can be setup by using the following commands (in this example we 515Soft limits can be setup by using the following commands (in this example we
412assume a soft limit of 256 megabytes) 516assume a soft limit of 256 MiB)
413 517
414# echo 256M > memory.soft_limit_in_bytes 518# echo 256M > memory.soft_limit_in_bytes
415 519
@@ -445,7 +549,7 @@ Note: Charges are moved only when you move mm->owner, IOW, a leader of a thread
445Note: If we cannot find enough space for the task in the destination cgroup, we 549Note: If we cannot find enough space for the task in the destination cgroup, we
446 try to make space by reclaiming memory. Task migration may fail if we 550 try to make space by reclaiming memory. Task migration may fail if we
447 cannot make enough space. 551 cannot make enough space.
448Note: It can take several seconds if you move charges in giga bytes order. 552Note: It can take several seconds if you move charges much.
449 553
450And if you want disable it again: 554And if you want disable it again:
451 555
@@ -465,7 +569,7 @@ memory cgroup.
465 | enable Swap Extension(see 2.4) to enable move of swap charges. 569 | enable Swap Extension(see 2.4) to enable move of swap charges.
466 -----+------------------------------------------------------------------------ 570 -----+------------------------------------------------------------------------
467 1 | A charge of file pages(normal file, tmpfs file(e.g. ipc shared memory) 571 1 | A charge of file pages(normal file, tmpfs file(e.g. ipc shared memory)
468 | and swaps of tmpfs file) mmaped by the target task. Unlike the case of 572 | and swaps of tmpfs file) mmapped by the target task. Unlike the case of
469 | anonymous pages, file pages(and swaps) in the range mmapped by the task 573 | anonymous pages, file pages(and swaps) in the range mmapped by the task
470 | will be moved even if the task hasn't done page fault, i.e. they might 574 | will be moved even if the task hasn't done page fault, i.e. they might
471 | not be the task's "RSS", but other task's "RSS" that maps the same file. 575 | not be the task's "RSS", but other task's "RSS" that maps the same file.
@@ -482,15 +586,15 @@ memory cgroup.
482 586
4839. Memory thresholds 5879. Memory thresholds
484 588
485Memory controler implements memory thresholds using cgroups notification 589Memory cgroup implements memory thresholds using cgroups notification
486API (see cgroups.txt). It allows to register multiple memory and memsw 590API (see cgroups.txt). It allows to register multiple memory and memsw
487thresholds and gets notifications when it crosses. 591thresholds and gets notifications when it crosses.
488 592
489To register a threshold application need: 593To register a threshold application need:
490 - create an eventfd using eventfd(2); 594- create an eventfd using eventfd(2);
491 - open memory.usage_in_bytes or memory.memsw.usage_in_bytes; 595- open memory.usage_in_bytes or memory.memsw.usage_in_bytes;
492 - write string like "<event_fd> <memory.usage_in_bytes> <threshold>" to 596- write string like "<event_fd> <fd of memory.usage_in_bytes> <threshold>" to
493 cgroup.event_control. 597 cgroup.event_control.
494 598
495Application will be notified through eventfd when memory usage crosses 599Application will be notified through eventfd when memory usage crosses
496threshold in any direction. 600threshold in any direction.
@@ -501,27 +605,28 @@ It's applicable for root and non-root cgroup.
501 605
502memory.oom_control file is for OOM notification and other controls. 606memory.oom_control file is for OOM notification and other controls.
503 607
504Memory controler implements oom notifier using cgroup notification 608Memory cgroup implements OOM notifier using cgroup notification
505API (See cgroups.txt). It allows to register multiple oom notification 609API (See cgroups.txt). It allows to register multiple OOM notification
506delivery and gets notification when oom happens. 610delivery and gets notification when OOM happens.
507 611
508To register a notifier, application need: 612To register a notifier, application need:
509 - create an eventfd using eventfd(2) 613 - create an eventfd using eventfd(2)
510 - open memory.oom_control file 614 - open memory.oom_control file
511 - write string like "<event_fd> <memory.oom_control>" to cgroup.event_control 615 - write string like "<event_fd> <fd of memory.oom_control>" to
616 cgroup.event_control
512 617
513Application will be notifier through eventfd when oom happens. 618Application will be notified through eventfd when OOM happens.
514OOM notification doesn't work for root cgroup. 619OOM notification doesn't work for root cgroup.
515 620
516You can disable oom-killer by writing "1" to memory.oom_control file. 621You can disable OOM-killer by writing "1" to memory.oom_control file, as:
517As. 622
518 #echo 1 > memory.oom_control 623 #echo 1 > memory.oom_control
519 624
520This operation is only allowed to the top cgroup of subhierarchy. 625This operation is only allowed to the top cgroup of sub-hierarchy.
521If oom-killer is disabled, tasks under cgroup will hang/sleep 626If OOM-killer is disabled, tasks under cgroup will hang/sleep
522in memcg's oom-waitq when they request accountable memory. 627in memory cgroup's OOM-waitqueue when they request accountable memory.
523 628
524For running them, you have to relax the memcg's oom sitaution by 629For running them, you have to relax the memory cgroup's OOM status by
525 * enlarge limit or reduce usage. 630 * enlarge limit or reduce usage.
526To reduce usage, 631To reduce usage,
527 * kill some tasks. 632 * kill some tasks.
@@ -532,7 +637,7 @@ Then, stopped tasks will work again.
532 637
533At reading, current status of OOM is shown. 638At reading, current status of OOM is shown.
534 oom_kill_disable 0 or 1 (if 1, oom-killer is disabled) 639 oom_kill_disable 0 or 1 (if 1, oom-killer is disabled)
535 under_oom 0 or 1 (if 1, the memcg is under OOM,tasks may 640 under_oom 0 or 1 (if 1, the memory cgroup is under OOM, tasks may
536 be stopped.) 641 be stopped.)
537 642
53811. TODO 64311. TODO