aboutsummaryrefslogtreecommitdiffstats
path: root/Documentation/cgroups
diff options
context:
space:
mode:
authorLinus Torvalds <torvalds@linux-foundation.org>2014-06-09 18:03:33 -0400
committerLinus Torvalds <torvalds@linux-foundation.org>2014-06-09 18:03:33 -0400
commit14208b0ec56919f5333dd654b1a7d10765d0ad05 (patch)
tree474b46c351efced45925d15dc2e0049c49784716 /Documentation/cgroups
parent6ea4fa70e4af0da8b133b246458fb789d8cb3985 (diff)
parentc731ae1d0f02a300697a8b1564780ad28a6c2013 (diff)
Merge branch 'for-3.16' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup
Pull cgroup updates from Tejun Heo: "A lot of activities on cgroup side. Heavy restructuring including locking simplification took place to improve the code base and enable implementation of the unified hierarchy, which currently exists behind a __DEVEL__ mount option. The core support is mostly complete but individual controllers need further work. To explain the design and rationales of the the unified hierarchy Documentation/cgroups/unified-hierarchy.txt is added. Another notable change is css (cgroup_subsys_state - what each controller uses to identify and interact with a cgroup) iteration update. This is part of continuing updates on css object lifetime and visibility. cgroup started with reference count draining on removal way back and is now reaching a point where csses behave and are iterated like normal refcnted objects albeit with some complexities to allow distinguishing the state where they're being deleted. The css iteration update isn't taken advantage of yet but is planned to be used to simplify memcg significantly" * 'for-3.16' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup: (77 commits) cgroup: disallow disabled controllers on the default hierarchy cgroup: don't destroy the default root cgroup: disallow debug controller on the default hierarchy cgroup: clean up MAINTAINERS entries cgroup: implement css_tryget() device_cgroup: use css_has_online_children() instead of has_children() cgroup: convert cgroup_has_live_children() into css_has_online_children() cgroup: use CSS_ONLINE instead of CGRP_DEAD cgroup: iterate cgroup_subsys_states directly cgroup: introduce CSS_RELEASED and reduce css iteration fallback window cgroup: move cgroup->serial_nr into cgroup_subsys_state cgroup: link all cgroup_subsys_states in their sibling lists cgroup: move cgroup->sibling and ->children into cgroup_subsys_state cgroup: remove cgroup->parent device_cgroup: remove direct access to cgroup->children memcg: update memcg_has_children() to use css_next_child() memcg: remove tasks/children test from mem_cgroup_force_empty() cgroup: remove css_parent() cgroup: skip refcnting on normal root csses and cgrp_dfl_root self css cgroup: use cgroup->self.refcnt for cgroup refcnting ...
Diffstat (limited to 'Documentation/cgroups')
-rw-r--r--Documentation/cgroups/memory.txt6
-rw-r--r--Documentation/cgroups/unified-hierarchy.txt359
2 files changed, 360 insertions, 5 deletions
diff --git a/Documentation/cgroups/memory.txt b/Documentation/cgroups/memory.txt
index b3429aec444c..02ab997a1ed2 100644
--- a/Documentation/cgroups/memory.txt
+++ b/Documentation/cgroups/memory.txt
@@ -458,15 +458,11 @@ About use_hierarchy, see Section 6.
458 458
4595.1 force_empty 4595.1 force_empty
460 memory.force_empty interface is provided to make cgroup's memory usage empty. 460 memory.force_empty interface is provided to make cgroup's memory usage empty.
461 You can use this interface only when the cgroup has no tasks.
462 When writing anything to this 461 When writing anything to this
463 462
464 # echo 0 > memory.force_empty 463 # echo 0 > memory.force_empty
465 464
466 Almost all pages tracked by this memory cgroup will be unmapped and freed. 465 the cgroup will be reclaimed and as many pages reclaimed as possible.
467 Some pages cannot be freed because they are locked or in-use. Such pages are
468 moved to parent (if use_hierarchy==1) or root (if use_hierarchy==0) and this
469 cgroup will be empty.
470 466
471 The typical use case for this interface is before calling rmdir(). 467 The typical use case for this interface is before calling rmdir().
472 Because rmdir() moves all pages to parent, some out-of-use page caches can be 468 Because rmdir() moves all pages to parent, some out-of-use page caches can be
diff --git a/Documentation/cgroups/unified-hierarchy.txt b/Documentation/cgroups/unified-hierarchy.txt
new file mode 100644
index 000000000000..324b182e6000
--- /dev/null
+++ b/Documentation/cgroups/unified-hierarchy.txt
@@ -0,0 +1,359 @@
1
2Cgroup unified hierarchy
3
4April, 2014 Tejun Heo <tj@kernel.org>
5
6This document describes the changes made by unified hierarchy and
7their rationales. It will eventually be merged into the main cgroup
8documentation.
9
10CONTENTS
11
121. Background
132. Basic Operation
14 2-1. Mounting
15 2-2. cgroup.subtree_control
16 2-3. cgroup.controllers
173. Structural Constraints
18 3-1. Top-down
19 3-2. No internal tasks
204. Other Changes
21 4-1. [Un]populated Notification
22 4-2. Other Core Changes
23 4-3. Per-Controller Changes
24 4-3-1. blkio
25 4-3-2. cpuset
26 4-3-3. memory
275. Planned Changes
28 5-1. CAP for resource control
29
30
311. Background
32
33cgroup allows an arbitrary number of hierarchies and each hierarchy
34can host any number of controllers. While this seems to provide a
35high level of flexibility, it isn't quite useful in practice.
36
37For example, as there is only one instance of each controller, utility
38type controllers such as freezer which can be useful in all
39hierarchies can only be used in one. The issue is exacerbated by the
40fact that controllers can't be moved around once hierarchies are
41populated. Another issue is that all controllers bound to a hierarchy
42are forced to have exactly the same view of the hierarchy. It isn't
43possible to vary the granularity depending on the specific controller.
44
45In practice, these issues heavily limit which controllers can be put
46on the same hierarchy and most configurations resort to putting each
47controller on its own hierarchy. Only closely related ones, such as
48the cpu and cpuacct controllers, make sense to put on the same
49hierarchy. This often means that userland ends up managing multiple
50similar hierarchies repeating the same steps on each hierarchy
51whenever a hierarchy management operation is necessary.
52
53Unfortunately, support for multiple hierarchies comes at a steep cost.
54Internal implementation in cgroup core proper is dazzlingly
55complicated but more importantly the support for multiple hierarchies
56restricts how cgroup is used in general and what controllers can do.
57
58There's no limit on how many hierarchies there may be, which means
59that a task's cgroup membership can't be described in finite length.
60The key may contain any varying number of entries and is unlimited in
61length, which makes it highly awkward to handle and leads to addition
62of controllers which exist only to identify membership, which in turn
63exacerbates the original problem.
64
65Also, as a controller can't have any expectation regarding what shape
66of hierarchies other controllers would be on, each controller has to
67assume that all other controllers are operating on completely
68orthogonal hierarchies. This makes it impossible, or at least very
69cumbersome, for controllers to cooperate with each other.
70
71In most use cases, putting controllers on hierarchies which are
72completely orthogonal to each other isn't necessary. What usually is
73called for is the ability to have differing levels of granularity
74depending on the specific controller. In other words, hierarchy may
75be collapsed from leaf towards root when viewed from specific
76controllers. For example, a given configuration might not care about
77how memory is distributed beyond a certain level while still wanting
78to control how CPU cycles are distributed.
79
80Unified hierarchy is the next version of cgroup interface. It aims to
81address the aforementioned issues by having more structure while
82retaining enough flexibility for most use cases. Various other
83general and controller-specific interface issues are also addressed in
84the process.
85
86
872. Basic Operation
88
892-1. Mounting
90
91Currently, unified hierarchy can be mounted with the following mount
92command. Note that this is still under development and scheduled to
93change soon.
94
95 mount -t cgroup -o __DEVEL__sane_behavior cgroup $MOUNT_POINT
96
97All controllers which are not bound to other hierarchies are
98automatically bound to unified hierarchy and show up at the root of
99it. Controllers which are enabled only in the root of unified
100hierarchy can be bound to other hierarchies at any time. This allows
101mixing unified hierarchy with the traditional multiple hierarchies in
102a fully backward compatible way.
103
104
1052-2. cgroup.subtree_control
106
107All cgroups on unified hierarchy have a "cgroup.subtree_control" file
108which governs which controllers are enabled on the children of the
109cgroup. Let's assume a hierarchy like the following.
110
111 root - A - B - C
112 \ D
113
114root's "cgroup.subtree_control" file determines which controllers are
115enabled on A. A's on B. B's on C and D. This coincides with the
116fact that controllers on the immediate sub-level are used to
117distribute the resources of the parent. In fact, it's natural to
118assume that resource control knobs of a child belong to its parent.
119Enabling a controller in a "cgroup.subtree_control" file declares that
120distribution of the respective resources of the cgroup will be
121controlled. Note that this means that controller enable states are
122shared among siblings.
123
124When read, the file contains a space-separated list of currently
125enabled controllers. A write to the file should contain a
126space-separated list of controllers with '+' or '-' prefixed (without
127the quotes). Controllers prefixed with '+' are enabled and '-'
128disabled. If a controller is listed multiple times, the last entry
129wins. The specific operations are executed atomically - either all
130succeed or fail.
131
132
1332-3. cgroup.controllers
134
135Read-only "cgroup.controllers" file contains a space-separated list of
136controllers which can be enabled in the cgroup's
137"cgroup.subtree_control" file.
138
139In the root cgroup, this lists controllers which are not bound to
140other hierarchies and the content changes as controllers are bound to
141and unbound from other hierarchies.
142
143In non-root cgroups, the content of this file equals that of the
144parent's "cgroup.subtree_control" file as only controllers enabled
145from the parent can be used in its children.
146
147
1483. Structural Constraints
149
1503-1. Top-down
151
152As it doesn't make sense to nest control of an uncontrolled resource,
153all non-root "cgroup.subtree_control" files can only contain
154controllers which are enabled in the parent's "cgroup.subtree_control"
155file. A controller can be enabled only if the parent has the
156controller enabled and a controller can't be disabled if one or more
157children have it enabled.
158
159
1603-2. No internal tasks
161
162One long-standing issue that cgroup faces is the competition between
163tasks belonging to the parent cgroup and its children cgroups. This
164is inherently nasty as two different types of entities compete and
165there is no agreed-upon obvious way to handle it. Different
166controllers are doing different things.
167
168The cpu controller considers tasks and cgroups as equivalents and maps
169nice levels to cgroup weights. This works for some cases but falls
170flat when children should be allocated specific ratios of CPU cycles
171and the number of internal tasks fluctuates - the ratios constantly
172change as the number of competing entities fluctuates. There also are
173other issues. The mapping from nice level to weight isn't obvious or
174universal, and there are various other knobs which simply aren't
175available for tasks.
176
177The blkio controller implicitly creates a hidden leaf node for each
178cgroup to host the tasks. The hidden leaf has its own copies of all
179the knobs with "leaf_" prefixed. While this allows equivalent control
180over internal tasks, it's with serious drawbacks. It always adds an
181extra layer of nesting which may not be necessary, makes the interface
182messy and significantly complicates the implementation.
183
184The memory controller currently doesn't have a way to control what
185happens between internal tasks and child cgroups and the behavior is
186not clearly defined. There have been attempts to add ad-hoc behaviors
187and knobs to tailor the behavior to specific workloads. Continuing
188this direction will lead to problems which will be extremely difficult
189to resolve in the long term.
190
191Multiple controllers struggle with internal tasks and came up with
192different ways to deal with it; unfortunately, all the approaches in
193use now are severely flawed and, furthermore, the widely different
194behaviors make cgroup as whole highly inconsistent.
195
196It is clear that this is something which needs to be addressed from
197cgroup core proper in a uniform way so that controllers don't need to
198worry about it and cgroup as a whole shows a consistent and logical
199behavior. To achieve that, unified hierarchy enforces the following
200structural constraint:
201
202 Except for the root, only cgroups which don't contain any task may
203 have controllers enabled in their "cgroup.subtree_control" files.
204
205Combined with other properties, this guarantees that, when a
206controller is looking at the part of the hierarchy which has it
207enabled, tasks are always only on the leaves. This rules out
208situations where child cgroups compete against internal tasks of the
209parent.
210
211There are two things to note. Firstly, the root cgroup is exempt from
212the restriction. Root contains tasks and anonymous resource
213consumption which can't be associated with any other cgroup and
214requires special treatment from most controllers. How resource
215consumption in the root cgroup is governed is up to each controller.
216
217Secondly, the restriction doesn't take effect if there is no enabled
218controller in the cgroup's "cgroup.subtree_control" file. This is
219important as otherwise it wouldn't be possible to create children of a
220populated cgroup. To control resource distribution of a cgroup, the
221cgroup must create children and transfer all its tasks to the children
222before enabling controllers in its "cgroup.subtree_control" file.
223
224
2254. Other Changes
226
2274-1. [Un]populated Notification
228
229cgroup users often need a way to determine when a cgroup's
230subhierarchy becomes empty so that it can be cleaned up. cgroup
231currently provides release_agent for it; unfortunately, this mechanism
232is riddled with issues.
233
234- It delivers events by forking and execing a userland binary
235 specified as the release_agent. This is a long deprecated method of
236 notification delivery. It's extremely heavy, slow and cumbersome to
237 integrate with larger infrastructure.
238
239- There is single monitoring point at the root. There's no way to
240 delegate management of a subtree.
241
242- The event isn't recursive. It triggers when a cgroup doesn't have
243 any tasks or child cgroups. Events for internal nodes trigger only
244 after all children are removed. This again makes it impossible to
245 delegate management of a subtree.
246
247- Events are filtered from the kernel side. A "notify_on_release"
248 file is used to subscribe to or suppress release events. This is
249 unnecessarily complicated and probably done this way because event
250 delivery itself was expensive.
251
252Unified hierarchy implements an interface file "cgroup.populated"
253which can be used to monitor whether the cgroup's subhierarchy has
254tasks in it or not. Its value is 0 if there is no task in the cgroup
255and its descendants; otherwise, 1. poll and [id]notify events are
256triggered when the value changes.
257
258This is significantly lighter and simpler and trivially allows
259delegating management of subhierarchy - subhierarchy monitoring can
260block further propagation simply by putting itself or another process
261in the subhierarchy and monitor events that it's interested in from
262there without interfering with monitoring higher in the tree.
263
264In unified hierarchy, the release_agent mechanism is no longer
265supported and the interface files "release_agent" and
266"notify_on_release" do not exist.
267
268
2694-2. Other Core Changes
270
271- None of the mount options is allowed.
272
273- remount is disallowed.
274
275- rename(2) is disallowed.
276
277- The "tasks" file is removed. Everything should at process
278 granularity. Use the "cgroup.procs" file instead.
279
280- The "cgroup.procs" file is not sorted. pids will be unique unless
281 they got recycled in-between reads.
282
283- The "cgroup.clone_children" file is removed.
284
285
2864-3. Per-Controller Changes
287
2884-3-1. blkio
289
290- blk-throttle becomes properly hierarchical.
291
292
2934-3-2. cpuset
294
295- Tasks are kept in empty cpusets after hotplug and take on the masks
296 of the nearest non-empty ancestor, instead of being moved to it.
297
298- A task can be moved into an empty cpuset, and again it takes on the
299 masks of the nearest non-empty ancestor.
300
301
3024-3-3. memory
303
304- use_hierarchy is on by default and the cgroup file for the flag is
305 not created.
306
307
3085. Planned Changes
309
3105-1. CAP for resource control
311
312Unified hierarchy will require one of the capabilities(7), which is
313yet to be decided, for all resource control related knobs. Process
314organization operations - creation of sub-cgroups and migration of
315processes in sub-hierarchies may be delegated by changing the
316ownership and/or permissions on the cgroup directory and
317"cgroup.procs" interface file; however, all operations which affect
318resource control - writes to a "cgroup.subtree_control" file or any
319controller-specific knobs - will require an explicit CAP privilege.
320
321This, in part, is to prevent the cgroup interface from being
322inadvertently promoted to programmable API used by non-privileged
323binaries. cgroup exposes various aspects of the system in ways which
324aren't properly abstracted for direct consumption by regular programs.
325This is an administration interface much closer to sysctl knobs than
326system calls. Even the basic access model, being filesystem path
327based, isn't suitable for direct consumption. There's no way to
328access "my cgroup" in a race-free way or make multiple operations
329atomic against migration to another cgroup.
330
331Another aspect is that, for better or for worse, the cgroup interface
332goes through far less scrutiny than regular interfaces for
333unprivileged userland. The upside is that cgroup is able to expose
334useful features which may not be suitable for general consumption in a
335reasonable time frame. It provides a relatively short path between
336internal details and userland-visible interface. Of course, this
337shortcut comes with high risk. We go through what we go through for
338general kernel APIs for good reasons. It may end up leaking internal
339details in a way which can exert significant pain by locking the
340kernel into a contract that can't be maintained in a reasonable
341manner.
342
343Also, due to the specific nature, cgroup and its controllers don't
344tend to attract attention from a wide scope of developers. cgroup's
345short history is already fraught with severely mis-designed
346interfaces, unnecessary commitments to and exposing of internal
347details, broken and dangerous implementations of various features.
348
349Keeping cgroup as an administration interface is both advantageous for
350its role and imperative given its nature. Some of the cgroup features
351may make sense for unprivileged access. If deemed justified, those
352must be further abstracted and implemented as a different interface,
353be it a system call or process-private filesystem, and survive through
354the scrutiny that any interface for general consumption is required to
355go through.
356
357Requiring CAP is not a complete solution but should serve as a
358significant deterrent against spraying cgroup usages in non-privileged
359programs.