aboutsummaryrefslogtreecommitdiffstats
path: root/Documentation/fault-injection/fault-injection.txt
diff options
context:
space:
mode:
authorAkinobu Mita <akinobu.mita@gmail.com>2006-12-08 05:39:42 -0500
committerLinus Torvalds <torvalds@woody.osdl.org>2006-12-08 11:29:02 -0500
commitde1ba09b214056365d9082982905b255caafb7a2 (patch)
tree6806f2abcfb9eee699424112a48c44edbbdff0c2 /Documentation/fault-injection/fault-injection.txt
parent4b3bb06bea649396490094780f90d315c152f6ab (diff)
[PATCH] fault injection: documentation and scripts
This patch set provides some fault-injection capabilities. - kmalloc() failures - alloc_pages() failures - disk IO errors We can see what really happens if those failures happen. In order to enable these fault-injection capabilities: 1. Enable relevant config options (CONFIG_FAILSLAB, CONFIG_PAGE_ALLOC, CONFIG_MAKE_REQUEST) and if you want to configure them via debugfs, enable CONFIG_FAULT_INJECTION_DEBUG_FS. 2. Build and boot with this kernel 3. Configure fault-injection capabilities behavior by boot option or debugfs - Boot option failslab= fail_page_alloc= fail_make_request= - Debugfs /debug/failslab/* /debug/fail_page_alloc/* /debug/fail_make_request/* Please refer to the Documentation/fault-injection/fault-injection.txt for details. 4. See what really happens. Signed-off-by: Akinobu Mita <akinobu.mita@gmail.com> Signed-off-by: Don Mullis <dwm@meer.net> Signed-off-by: Andrew Morton <akpm@osdl.org> Signed-off-by: Linus Torvalds <torvalds@osdl.org>
Diffstat (limited to 'Documentation/fault-injection/fault-injection.txt')
-rw-r--r--Documentation/fault-injection/fault-injection.txt223
1 files changed, 223 insertions, 0 deletions
diff --git a/Documentation/fault-injection/fault-injection.txt b/Documentation/fault-injection/fault-injection.txt
new file mode 100644
index 000000000000..260ce6c199ce
--- /dev/null
+++ b/Documentation/fault-injection/fault-injection.txt
@@ -0,0 +1,223 @@
1Fault injection capabilities infrastructure
2===========================================
3
4See also drivers/md/faulty.c and "every_nth" module option for scsi_debug.
5
6
7Available fault injection capabilities
8--------------------------------------
9
10o failslab
11
12 injects slab allocation failures. (kmalloc(), kmem_cache_alloc(), ...)
13
14o fail_page_alloc
15
16 injects page allocation failures. (alloc_pages(), get_free_pages(), ...)
17
18o fail_make_request
19
20 injects disk IO errors on permitted devices by
21 /sys/block/<device>/make-it-fail or
22 /sys/block/<device>/<partition>/make-it-fail. (generic_make_request())
23
24Configure fault-injection capabilities behavior
25-----------------------------------------------
26
27o debugfs entries
28
29fault-inject-debugfs kernel module provides some debugfs entries for runtime
30configuration of fault-injection capabilities.
31
32- /debug/*/probability:
33
34 likelihood of failure injection, in percent.
35 Format: <percent>
36
37 Note that one-failure-per-handred is a very high error rate
38 for some testcases. Please set probably=100 and configure
39 /debug/*/interval for such testcases.
40
41- /debug/*/interval:
42
43 specifies the interval between failures, for calls to
44 should_fail() that pass all the other tests.
45
46 Note that if you enable this, by setting interval>1, you will
47 probably want to set probability=100.
48
49- /debug/*/times:
50
51 specifies how many times failures may happen at most.
52 A value of -1 means "no limit".
53
54- /debug/*/space:
55
56 specifies an initial resource "budget", decremented by "size"
57 on each call to should_fail(,size). Failure injection is
58 suppressed until "space" reaches zero.
59
60- /debug/*/verbose
61
62 Format: { 0 | 1 | 2 }
63 specifies the verbosity of the messages when failure is injected.
64 We default to 0 (no extra messages), setting it to '1' will
65 print only to tell failure happened, '2' will print call trace too -
66 it is useful to debug the problems revealed by fault injection
67 capabilities.
68
69- /debug/*/task-filter:
70
71 Format: { 0 | 1 }
72 A value of '0' disables filtering by process (default).
73 Any positive value limits failures to only processes indicated by
74 /proc/<pid>/make-it-fail==1.
75
76- /debug/*/address-start:
77- /debug/*/address-end:
78
79 specifies the range of virtual addresses tested during
80 stacktrace walking. Failure is injected only if some caller
81 in the walked stacktrace lies within this range.
82 Default is [0,ULONG_MAX) (whole of virtual address space).
83
84- /debug/*/stacktrace-depth:
85
86 specifies the maximum stacktrace depth walked during search
87 for a caller within [address-start,address-end).
88
89- /debug/failslab/ignore-gfp-highmem:
90- /debug/fail_page_alloc/ignore-gfp-highmem:
91
92 Format: { 0 | 1 }
93 default is 0, setting it to '1' won't inject failures into
94 highmem/user allocations.
95
96- /debug/failslab/ignore-gfp-wait:
97- /debug/fail_page_alloc/ignore-gfp-wait:
98
99 Format: { 0 | 1 }
100 default is 0, setting it to '1' will inject failures
101 only into non-sleep allocations (GFP_ATOMIC allocations).
102
103o Boot option
104
105In order to inject faults while debugfs is not available (early boot time),
106use the boot option:
107
108 failslab=
109 fail_page_alloc=
110 fail_make_request=<interval>,<probability>,<space>,<times>
111
112How to add new fault injection capability
113-----------------------------------------
114
115o #include <linux/fault-inject.h>
116
117o define the fault attributes
118
119 DECLARE_FAULT_INJECTION(name);
120
121 Please see the definition of struct fault_attr in fault-inject.h
122 for details.
123
124o provide the way to configure fault attributes
125
126- boot option
127
128 If you need to enable the fault injection capability from boot time, you can
129 provide boot option to configure it. There is a helper function for it.
130
131 setup_fault_attr(attr, str);
132
133- debugfs entries
134
135 failslab, fail_page_alloc, and fail_make_request use this way.
136 There is a helper function for it.
137
138 init_fault_attr_entries(entries, attr, name);
139 void cleanup_fault_attr_entries(entries);
140
141- module parameters
142
143 If the scope of the fault injection capability is limited to a
144 single kernel module, it is better to provide module parameters to
145 configure the fault attributes.
146
147o add a hook to insert failures
148
149 should_fail() returns 1 when failures should happen.
150
151 should_fail(attr,size);
152
153Application Examples
154--------------------
155
156o inject slab allocation failures into module init/cleanup code
157
158------------------------------------------------------------------------------
159#!/bin/bash
160
161FAILCMD=Documentation/fault-injection/failcmd.sh
162BLACKLIST="root_plug evbug"
163
164FAILNAME=failslab
165echo Y > /debug/$FAILNAME/task-filter
166echo 10 > /debug/$FAILNAME/probability
167echo 100 > /debug/$FAILNAME/interval
168echo -1 > /debug/$FAILNAME/times
169echo 2 > /debug/$FAILNAME/verbose
170echo 1 > /debug/$FAILNAME/ignore-gfp-highmem
171echo 1 > /debug/$FAILNAME/ignore-gfp-wait
172
173blacklist()
174{
175 echo $BLACKLIST | grep $1 > /dev/null 2>&1
176}
177
178oops()
179{
180 dmesg | grep BUG > /dev/null 2>&1
181}
182
183find /lib/modules/`uname -r` -name '*.ko' -exec basename {} .ko \; |
184 while read i
185 do
186 oops && exit 1
187
188 if ! blacklist $i
189 then
190 echo inserting $i...
191 bash $FAILCMD modprobe $i
192 fi
193 done
194
195lsmod | awk '{ if ($3 == 0) { print $1 } }' |
196 while read i
197 do
198 oops && exit 1
199
200 if ! blacklist $i
201 then
202 echo removing $i...
203 bash $FAILCMD modprobe -r $i
204 fi
205 done
206
207------------------------------------------------------------------------------
208
209o inject slab allocation failures only for a specific module
210
211------------------------------------------------------------------------------
212#!/bin/bash
213
214FAILMOD=Documentation/fault-injection/failmodule.sh
215
216echo injecting errors into the module $1...
217
218modprobe $1
219bash $FAILMOD failslab $1 10
220echo 25 > /debug/failslab/probability
221
222------------------------------------------------------------------------------
223