diff options
author | Mauro Carvalho Chehab <mchehab@s-opensource.com> | 2016-09-21 08:51:11 -0400 |
---|---|---|
committer | Mauro Carvalho Chehab <mchehab@s-opensource.com> | 2016-10-24 06:12:35 -0400 |
commit | 9d85025b0418163fae079c9ba8f8445212de8568 (patch) | |
tree | 4629e2dedf4a9ed45a6855c129101f9b52138372 /Documentation/admin-guide/oops-tracing.rst | |
parent | 186128f75392f8478ad1b32a675627d738881ca4 (diff) |
docs-rst: create an user's manual book
Place README, REPORTING-BUGS, SecurityBugs and kernel-parameters
on an user's manual book.
As we'll be numbering the user's manual, remove the manual
numbering from SecurityBugs.
Signed-off-by: Mauro Carvalho Chehab <mchehab@s-opensource.com>
Diffstat (limited to 'Documentation/admin-guide/oops-tracing.rst')
-rw-r--r-- | Documentation/admin-guide/oops-tracing.rst | 300 |
1 files changed, 300 insertions, 0 deletions
diff --git a/Documentation/admin-guide/oops-tracing.rst b/Documentation/admin-guide/oops-tracing.rst new file mode 100644 index 000000000000..3e25ea7349ee --- /dev/null +++ b/Documentation/admin-guide/oops-tracing.rst | |||
@@ -0,0 +1,300 @@ | |||
1 | OOPS tracing | ||
2 | ============ | ||
3 | |||
4 | .. note:: | ||
5 | |||
6 | ``ksymoops`` is useless on 2.6 or upper. Please use the Oops in its original | ||
7 | format (from ``dmesg``, etc). Ignore any references in this or other docs to | ||
8 | "decoding the Oops" or "running it through ksymoops". | ||
9 | If you post an Oops from 2.6+ that has been run through ``ksymoops``, | ||
10 | people will just tell you to repost it. | ||
11 | |||
12 | Quick Summary | ||
13 | ------------- | ||
14 | |||
15 | Find the Oops and send it to the maintainer of the kernel area that seems to be | ||
16 | involved with the problem. Don't worry too much about getting the wrong person. | ||
17 | If you are unsure send it to the person responsible for the code relevant to | ||
18 | what you were doing. If it occurs repeatably try and describe how to recreate | ||
19 | it. That's worth even more than the oops. | ||
20 | |||
21 | If you are totally stumped as to whom to send the report, send it to | ||
22 | linux-kernel@vger.kernel.org. Thanks for your help in making Linux as | ||
23 | stable as humanly possible. | ||
24 | |||
25 | Where is the Oops? | ||
26 | ---------------------- | ||
27 | |||
28 | Normally the Oops text is read from the kernel buffers by klogd and | ||
29 | handed to ``syslogd`` which writes it to a syslog file, typically | ||
30 | ``/var/log/messages`` (depends on ``/etc/syslog.conf``). Sometimes ``klogd`` | ||
31 | dies, in which case you can run ``dmesg > file`` to read the data from the | ||
32 | kernel buffers and save it. Or you can ``cat /proc/kmsg > file``, however you | ||
33 | have to break in to stop the transfer, ``kmsg`` is a "never ending file". | ||
34 | If the machine has crashed so badly that you cannot enter commands or | ||
35 | the disk is not available then you have three options : | ||
36 | |||
37 | (1) Hand copy the text from the screen and type it in after the machine | ||
38 | has restarted. Messy but it is the only option if you have not | ||
39 | planned for a crash. Alternatively, you can take a picture of | ||
40 | the screen with a digital camera - not nice, but better than | ||
41 | nothing. If the messages scroll off the top of the console, you | ||
42 | may find that booting with a higher resolution (eg, ``vga=791``) | ||
43 | will allow you to read more of the text. (Caveat: This needs ``vesafb``, | ||
44 | so won't help for 'early' oopses) | ||
45 | |||
46 | (2) Boot with a serial console (see | ||
47 | :ref:`Documentation/serial-console.txt <serial_console>`), | ||
48 | run a null modem to a second machine and capture the output there | ||
49 | using your favourite communication program. Minicom works well. | ||
50 | |||
51 | (3) Use Kdump (see Documentation/kdump/kdump.txt), | ||
52 | extract the kernel ring buffer from old memory with using dmesg | ||
53 | gdbmacro in Documentation/kdump/gdbmacros.txt. | ||
54 | |||
55 | |||
56 | Full Information | ||
57 | ---------------- | ||
58 | |||
59 | .. note:: | ||
60 | |||
61 | the message from Linus below applies to 2.4 kernel. I have preserved it | ||
62 | for historical reasons, and because some of the information in it still | ||
63 | applies. Especially, please ignore any references to ksymoops. | ||
64 | |||
65 | :: | ||
66 | |||
67 | From: Linus Torvalds <torvalds@osdl.org> | ||
68 | |||
69 | How to track down an Oops.. [originally a mail to linux-kernel] | ||
70 | |||
71 | The main trick is having 5 years of experience with those pesky oops | ||
72 | messages ;-) | ||
73 | |||
74 | Actually, there are things you can do that make this easier. I have two | ||
75 | separate approaches:: | ||
76 | |||
77 | gdb /usr/src/linux/vmlinux | ||
78 | gdb> disassemble <offending_function> | ||
79 | |||
80 | That's the easy way to find the problem, at least if the bug-report is | ||
81 | well made (like this one was - run through ``ksymoops`` to get the | ||
82 | information of which function and the offset in the function that it | ||
83 | happened in). | ||
84 | |||
85 | Oh, it helps if the report happens on a kernel that is compiled with the | ||
86 | same compiler and similar setups. | ||
87 | |||
88 | The other thing to do is disassemble the "Code:" part of the bug report: | ||
89 | ksymoops will do this too with the correct tools, but if you don't have | ||
90 | the tools you can just do a silly program:: | ||
91 | |||
92 | char str[] = "\xXX\xXX\xXX..."; | ||
93 | main(){} | ||
94 | |||
95 | and compile it with ``gcc -g`` and then do ``disassemble str`` (where the ``XX`` | ||
96 | stuff are the values reported by the Oops - you can just cut-and-paste | ||
97 | and do a replace of spaces to ``\x`` - that's what I do, as I'm too lazy | ||
98 | to write a program to automate this all). | ||
99 | |||
100 | Alternatively, you can use the shell script in ``scripts/decodecode``. | ||
101 | Its usage is:: | ||
102 | |||
103 | decodecode < oops.txt | ||
104 | |||
105 | The hex bytes that follow "Code:" may (in some architectures) have a series | ||
106 | of bytes that precede the current instruction pointer as well as bytes at and | ||
107 | following the current instruction pointer. In some cases, one instruction | ||
108 | byte or word is surrounded by ``<>`` or ``()``, as in ``<86>`` or ``(f00d)``. | ||
109 | These ``<>`` or ``()`` markings indicate the current instruction pointer. | ||
110 | |||
111 | Example from i386, split into multiple lines for readability:: | ||
112 | |||
113 | Code: f9 0f 8d f9 00 00 00 8d 42 0c e8 dd 26 11 c7 a1 60 ea 2b f9 8b 50 08 a1 | ||
114 | 64 ea 2b f9 8d 34 82 8b 1e 85 db 74 6d 8b 15 60 ea 2b f9 <8b> 43 04 39 42 54 | ||
115 | 7e 04 40 89 42 54 8b 43 04 3b 05 00 f6 52 c0 | ||
116 | |||
117 | Finally, if you want to see where the code comes from, you can do:: | ||
118 | |||
119 | cd /usr/src/linux | ||
120 | make fs/buffer.s # or whatever file the bug happened in | ||
121 | |||
122 | and then you get a better idea of what happens than with the gdb | ||
123 | disassembly. | ||
124 | |||
125 | Now, the trick is just then to combine all the data you have: the C | ||
126 | sources (and general knowledge of what it **should** do), the assembly | ||
127 | listing and the code disassembly (and additionally the register dump you | ||
128 | also get from the "oops" message - that can be useful to see **what** the | ||
129 | corrupted pointers were, and when you have the assembler listing you can | ||
130 | also match the other registers to whatever C expressions they were used | ||
131 | for). | ||
132 | |||
133 | Essentially, you just look at what doesn't match (in this case it was the | ||
134 | "Code" disassembly that didn't match with what the compiler generated). | ||
135 | Then you need to find out **why** they don't match. Often it's simple - you | ||
136 | see that the code uses a NULL pointer and then you look at the code and | ||
137 | wonder how the NULL pointer got there, and if it's a valid thing to do | ||
138 | you just check against it.. | ||
139 | |||
140 | Now, if somebody gets the idea that this is time-consuming and requires | ||
141 | some small amount of concentration, you're right. Which is why I will | ||
142 | mostly just ignore any panic reports that don't have the symbol table | ||
143 | info etc looked up: it simply gets too hard to look it up (I have some | ||
144 | programs to search for specific patterns in the kernel code segment, and | ||
145 | sometimes I have been able to look up those kinds of panics too, but | ||
146 | that really requires pretty good knowledge of the kernel just to be able | ||
147 | to pick out the right sequences etc..) | ||
148 | |||
149 | **Sometimes** it happens that I just see the disassembled code sequence | ||
150 | from the panic, and I know immediately where it's coming from. That's when | ||
151 | I get worried that I've been doing this for too long ;-) | ||
152 | |||
153 | Linus | ||
154 | |||
155 | |||
156 | --------------------------------------------------------------------------- | ||
157 | |||
158 | Notes on Oops tracing with ``klogd`` | ||
159 | ------------------------------------ | ||
160 | |||
161 | In order to help Linus and the other kernel developers there has been | ||
162 | substantial support incorporated into ``klogd`` for processing protection | ||
163 | faults. In order to have full support for address resolution at least | ||
164 | version 1.3-pl3 of the ``sysklogd`` package should be used. | ||
165 | |||
166 | When a protection fault occurs the ``klogd`` daemon automatically | ||
167 | translates important addresses in the kernel log messages to their | ||
168 | symbolic equivalents. This translated kernel message is then | ||
169 | forwarded through whatever reporting mechanism ``klogd`` is using. The | ||
170 | protection fault message can be simply cut out of the message files | ||
171 | and forwarded to the kernel developers. | ||
172 | |||
173 | Two types of address resolution are performed by ``klogd``. The first is | ||
174 | static translation and the second is dynamic translation. Static | ||
175 | translation uses the System.map file in much the same manner that | ||
176 | ksymoops does. In order to do static translation the ``klogd`` daemon | ||
177 | must be able to find a system map file at daemon initialization time. | ||
178 | See the klogd man page for information on how ``klogd`` searches for map | ||
179 | files. | ||
180 | |||
181 | Dynamic address translation is important when kernel loadable modules | ||
182 | are being used. Since memory for kernel modules is allocated from the | ||
183 | kernel's dynamic memory pools there are no fixed locations for either | ||
184 | the start of the module or for functions and symbols in the module. | ||
185 | |||
186 | The kernel supports system calls which allow a program to determine | ||
187 | which modules are loaded and their location in memory. Using these | ||
188 | system calls the klogd daemon builds a symbol table which can be used | ||
189 | to debug a protection fault which occurs in a loadable kernel module. | ||
190 | |||
191 | At the very minimum klogd will provide the name of the module which | ||
192 | generated the protection fault. There may be additional symbolic | ||
193 | information available if the developer of the loadable module chose to | ||
194 | export symbol information from the module. | ||
195 | |||
196 | Since the kernel module environment can be dynamic there must be a | ||
197 | mechanism for notifying the ``klogd`` daemon when a change in module | ||
198 | environment occurs. There are command line options available which | ||
199 | allow klogd to signal the currently executing daemon that symbol | ||
200 | information should be refreshed. See the ``klogd`` manual page for more | ||
201 | information. | ||
202 | |||
203 | A patch is included with the sysklogd distribution which modifies the | ||
204 | ``modules-2.0.0`` package to automatically signal klogd whenever a module | ||
205 | is loaded or unloaded. Applying this patch provides essentially | ||
206 | seamless support for debugging protection faults which occur with | ||
207 | kernel loadable modules. | ||
208 | |||
209 | The following is an example of a protection fault in a loadable module | ||
210 | processed by ``klogd``:: | ||
211 | |||
212 | Aug 29 09:51:01 blizard kernel: Unable to handle kernel paging request at virtual address f15e97cc | ||
213 | Aug 29 09:51:01 blizard kernel: current->tss.cr3 = 0062d000, %cr3 = 0062d000 | ||
214 | Aug 29 09:51:01 blizard kernel: *pde = 00000000 | ||
215 | Aug 29 09:51:01 blizard kernel: Oops: 0002 | ||
216 | Aug 29 09:51:01 blizard kernel: CPU: 0 | ||
217 | Aug 29 09:51:01 blizard kernel: EIP: 0010:[oops:_oops+16/3868] | ||
218 | Aug 29 09:51:01 blizard kernel: EFLAGS: 00010212 | ||
219 | Aug 29 09:51:01 blizard kernel: eax: 315e97cc ebx: 003a6f80 ecx: 001be77b edx: 00237c0c | ||
220 | Aug 29 09:51:01 blizard kernel: esi: 00000000 edi: bffffdb3 ebp: 00589f90 esp: 00589f8c | ||
221 | Aug 29 09:51:01 blizard kernel: ds: 0018 es: 0018 fs: 002b gs: 002b ss: 0018 | ||
222 | Aug 29 09:51:01 blizard kernel: Process oops_test (pid: 3374, process nr: 21, stackpage=00589000) | ||
223 | Aug 29 09:51:01 blizard kernel: Stack: 315e97cc 00589f98 0100b0b4 bffffed4 0012e38e 00240c64 003a6f80 00000001 | ||
224 | Aug 29 09:51:01 blizard kernel: 00000000 00237810 bfffff00 0010a7fa 00000003 00000001 00000000 bfffff00 | ||
225 | Aug 29 09:51:01 blizard kernel: bffffdb3 bffffed4 ffffffda 0000002b 0007002b 0000002b 0000002b 00000036 | ||
226 | Aug 29 09:51:01 blizard kernel: Call Trace: [oops:_oops_ioctl+48/80] [_sys_ioctl+254/272] [_system_call+82/128] | ||
227 | Aug 29 09:51:01 blizard kernel: Code: c7 00 05 00 00 00 eb 08 90 90 90 90 90 90 90 90 89 ec 5d c3 | ||
228 | |||
229 | --------------------------------------------------------------------------- | ||
230 | |||
231 | :: | ||
232 | |||
233 | Dr. G.W. Wettstein Oncology Research Div. Computing Facility | ||
234 | Roger Maris Cancer Center INTERNET: greg@wind.rmcc.com | ||
235 | 820 4th St. N. | ||
236 | Fargo, ND 58122 | ||
237 | Phone: 701-234-7556 | ||
238 | |||
239 | |||
240 | --------------------------------------------------------------------------- | ||
241 | |||
242 | Tainted kernels | ||
243 | --------------- | ||
244 | |||
245 | Some oops reports contain the string **'Tainted: '** after the program | ||
246 | counter. This indicates that the kernel has been tainted by some | ||
247 | mechanism. The string is followed by a series of position-sensitive | ||
248 | characters, each representing a particular tainted value. | ||
249 | |||
250 | 1) 'G' if all modules loaded have a GPL or compatible license, 'P' if | ||
251 | any proprietary module has been loaded. Modules without a | ||
252 | MODULE_LICENSE or with a MODULE_LICENSE that is not recognised by | ||
253 | insmod as GPL compatible are assumed to be proprietary. | ||
254 | |||
255 | 2) ``F`` if any module was force loaded by ``insmod -f``, ``' '`` if all | ||
256 | modules were loaded normally. | ||
257 | |||
258 | 3) ``S`` if the oops occurred on an SMP kernel running on hardware that | ||
259 | hasn't been certified as safe to run multiprocessor. | ||
260 | Currently this occurs only on various Athlons that are not | ||
261 | SMP capable. | ||
262 | |||
263 | 4) ``R`` if a module was force unloaded by ``rmmod -f``, ``' '`` if all | ||
264 | modules were unloaded normally. | ||
265 | |||
266 | 5) ``M`` if any processor has reported a Machine Check Exception, | ||
267 | ``' '`` if no Machine Check Exceptions have occurred. | ||
268 | |||
269 | 6) ``B`` if a page-release function has found a bad page reference or | ||
270 | some unexpected page flags. | ||
271 | |||
272 | 7) ``U`` if a user or user application specifically requested that the | ||
273 | Tainted flag be set, ``' '`` otherwise. | ||
274 | |||
275 | 8) ``D`` if the kernel has died recently, i.e. there was an OOPS or BUG. | ||
276 | |||
277 | 9) ``A`` if the ACPI table has been overridden. | ||
278 | |||
279 | 10) ``W`` if a warning has previously been issued by the kernel. | ||
280 | (Though some warnings may set more specific taint flags.) | ||
281 | |||
282 | 11) ``C`` if a staging driver has been loaded. | ||
283 | |||
284 | 12) ``I`` if the kernel is working around a severe bug in the platform | ||
285 | firmware (BIOS or similar). | ||
286 | |||
287 | 13) ``O`` if an externally-built ("out-of-tree") module has been loaded. | ||
288 | |||
289 | 14) ``E`` if an unsigned module has been loaded in a kernel supporting | ||
290 | module signature. | ||
291 | |||
292 | 15) ``L`` if a soft lockup has previously occurred on the system. | ||
293 | |||
294 | 16) ``K`` if the kernel has been live patched. | ||
295 | |||
296 | The primary reason for the **'Tainted: '** string is to tell kernel | ||
297 | debuggers if this is a clean kernel or if anything unusual has | ||
298 | occurred. Tainting is permanent: even if an offending module is | ||
299 | unloaded, the tainted value remains to indicate that the kernel is not | ||
300 | trustworthy. | ||