1 files changed, 229 insertions, 0 deletions

diff --git a/Documentation/oops-tracing.txt b/Documentation/oops-tracing.txt
new file mode 100644
index 000000000000..da711028e5f7
--- /dev/null
+++ b/Documentation/oops-tracing.txt
@@ -0,0 +1,229 @@
+NOTE: ksymoops is useless on 2.6.  Please use the Oops in its original format
+(from dmesg, etc).  Ignore any references in this or other docs to "decoding
+the Oops" or "running it through ksymoops".  If you post an Oops fron 2.6 that
+has been run through ksymoops, people will just tell you to repost it.
+
+Quick Summary
+-------------
+
+Find the Oops and send it to the maintainer of the kernel area that seems to be
+involved with the problem.  Don't worry too much about getting the wrong person.
+If you are unsure send it to the person responsible for the code relevant to
+what you were doing.  If it occurs repeatably try and describe how to recreate
+it.  That's worth even more than the oops.
+
+If you are totally stumped as to whom to send the report, send it to 
+linux-kernel@vger.kernel.org. Thanks for your help in making Linux as
+stable as humanly possible.
+
+Where is the Oops?
+----------------------
+
+Normally the Oops text is read from the kernel buffers by klogd and
+handed to syslogd which writes it to a syslog file, typically
+/var/log/messages (depends on /etc/syslog.conf).  Sometimes klogd dies,
+in which case you can run dmesg > file to read the data from the kernel
+buffers and save it.  Or you can cat /proc/kmsg > file, however you
+have to break in to stop the transfer, kmsg is a "never ending file".
+If the machine has crashed so badly that you cannot enter commands or
+the disk is not available then you have three options :-
+
+(1) Hand copy the text from the screen and type it in after the machine
+    has restarted.  Messy but it is the only option if you have not
+    planned for a crash.
+
+(2) Boot with a serial console (see Documentation/serial-console.txt),
+    run a null modem to a second machine and capture the output there
+    using your favourite communication program.  Minicom works well.
+
+(3) Patch the kernel with one of the crash dump patches.  These save
+    data to a floppy disk or video rom or a swap partition.  None of
+    these are standard kernel patches so you have to find and apply
+    them yourself.  Search kernel archives for kmsgdump, lkcd and
+    oops+smram.
+
+
+Full Information
+----------------
+
+NOTE: the message from Linus below applies to 2.4 kernel.  I have preserved it
+for historical reasons, and because some of the information in it still
+applies.  Especially, please ignore any references to ksymoops. 
+
+From: Linus Torvalds <torvalds@osdl.org>
+
+How to track down an Oops.. [originally a mail to linux-kernel]
+
+The main trick is having 5 years of experience with those pesky oops 
+messages ;-)
+
+Actually, there are things you can do that make this easier. I have two 
+separate approaches:
+
+        gdb /usr/src/linux/vmlinux
+        gdb> disassemble <offending_function>
+
+That's the easy way to find the problem, at least if the bug-report is 
+well made (like this one was - run through ksymoops to get the 
+information of which function and the offset in the function that it 
+happened in).
+
+Oh, it helps if the report happens on a kernel that is compiled with the 
+same compiler and similar setups.
+
+The other thing to do is disassemble the "Code:" part of the bug report: 
+ksymoops will do this too with the correct tools, but if you don't have
+the tools you can just do a silly program:
+
+        char str[] = "\xXX\xXX\xXX...";
+        main(){}
+
+and compile it with gcc -g and then do "disassemble str" (where the "XX" 
+stuff are the values reported by the Oops - you can just cut-and-paste 
+and do a replace of spaces to "\x" - that's what I do, as I'm too lazy 
+to write a program to automate this all).
+
+Finally, if you want to see where the code comes from, you can do
+
+        cd /usr/src/linux
+        make fs/buffer.s        # or whatever file the bug happened in
+
+and then you get a better idea of what happens than with the gdb 
+disassembly.
+
+Now, the trick is just then to combine all the data you have: the C 
+sources (and general knowledge of what it _should_ do), the assembly 
+listing and the code disassembly (and additionally the register dump you 
+also get from the "oops" message - that can be useful to see _what_ the 
+corrupted pointers were, and when you have the assembler listing you can 
+also match the other registers to whatever C expressions they were used 
+for).
+
+Essentially, you just look at what doesn't match (in this case it was the 
+"Code" disassembly that didn't match with what the compiler generated). 
+Then you need to find out _why_ they don't match. Often it's simple - you 
+see that the code uses a NULL pointer and then you look at the code and 
+wonder how the NULL pointer got there, and if it's a valid thing to do 
+you just check against it..
+
+Now, if somebody gets the idea that this is time-consuming and requires 
+some small amount of concentration, you're right. Which is why I will 
+mostly just ignore any panic reports that don't have the symbol table 
+info etc looked up: it simply gets too hard to look it up (I have some 
+programs to search for specific patterns in the kernel code segment, and 
+sometimes I have been able to look up those kinds of panics too, but 
+that really requires pretty good knowledge of the kernel just to be able 
+to pick out the right sequences etc..)
+
+_Sometimes_ it happens that I just see the disassembled code sequence 
+from the panic, and I know immediately where it's coming from. That's when 
+I get worried that I've been doing this for too long ;-)
+
+                Linus
+
+
+---------------------------------------------------------------------------
+Notes on Oops tracing with klogd:
+
+In order to help Linus and the other kernel developers there has been
+substantial support incorporated into klogd for processing protection
+faults.  In order to have full support for address resolution at least
+version 1.3-pl3 of the sysklogd package should be used.
+
+When a protection fault occurs the klogd daemon automatically
+translates important addresses in the kernel log messages to their
+symbolic equivalents.  This translated kernel message is then
+forwarded through whatever reporting mechanism klogd is using.  The
+protection fault message can be simply cut out of the message files
+and forwarded to the kernel developers.
+
+Two types of address resolution are performed by klogd.  The first is
+static translation and the second is dynamic translation.  Static
+translation uses the System.map file in much the same manner that
+ksymoops does.  In order to do static translation the klogd daemon
+must be able to find a system map file at daemon initialization time.
+See the klogd man page for information on how klogd searches for map
+files.
+
+Dynamic address translation is important when kernel loadable modules
+are being used.  Since memory for kernel modules is allocated from the
+kernel's dynamic memory pools there are no fixed locations for either
+the start of the module or for functions and symbols in the module.
+
+The kernel supports system calls which allow a program to determine
+which modules are loaded and their location in memory.  Using these
+system calls the klogd daemon builds a symbol table which can be used
+to debug a protection fault which occurs in a loadable kernel module.
+
+At the very minimum klogd will provide the name of the module which
+generated the protection fault.  There may be additional symbolic
+information available if the developer of the loadable module chose to
+export symbol information from the module.
+
+Since the kernel module environment can be dynamic there must be a
+mechanism for notifying the klogd daemon when a change in module
+environment occurs.  There are command line options available which
+allow klogd to signal the currently executing daemon that symbol
+information should be refreshed.  See the klogd manual page for more
+information.
+
+A patch is included with the sysklogd distribution which modifies the
+modules-2.0.0 package to automatically signal klogd whenever a module
+is loaded or unloaded.  Applying this patch provides essentially
+seamless support for debugging protection faults which occur with
+kernel loadable modules.
+
+The following is an example of a protection fault in a loadable module
+processed by klogd:
+---------------------------------------------------------------------------
+Aug 29 09:51:01 blizard kernel: Unable to handle kernel paging request at virtual address f15e97cc
+Aug 29 09:51:01 blizard kernel: current->tss.cr3 = 0062d000, %cr3 = 0062d000
+Aug 29 09:51:01 blizard kernel: *pde = 00000000
+Aug 29 09:51:01 blizard kernel: Oops: 0002
+Aug 29 09:51:01 blizard kernel: CPU:    0
+Aug 29 09:51:01 blizard kernel: EIP:    0010:[oops:_oops+16/3868]
+Aug 29 09:51:01 blizard kernel: EFLAGS: 00010212
+Aug 29 09:51:01 blizard kernel: eax: 315e97cc   ebx: 003a6f80   ecx: 001be77b   edx: 00237c0c
+Aug 29 09:51:01 blizard kernel: esi: 00000000   edi: bffffdb3   ebp: 00589f90   esp: 00589f8c
+Aug 29 09:51:01 blizard kernel: ds: 0018   es: 0018   fs: 002b   gs: 002b   ss: 0018
+Aug 29 09:51:01 blizard kernel: Process oops_test (pid: 3374, process nr: 21, stackpage=00589000)
+Aug 29 09:51:01 blizard kernel: Stack: 315e97cc 00589f98 0100b0b4 bffffed4 0012e38e 00240c64 003a6f80 00000001 
+Aug 29 09:51:01 blizard kernel:        00000000 00237810 bfffff00 0010a7fa 00000003 00000001 00000000 bfffff00 
+Aug 29 09:51:01 blizard kernel:        bffffdb3 bffffed4 ffffffda 0000002b 0007002b 0000002b 0000002b 00000036 
+Aug 29 09:51:01 blizard kernel: Call Trace: [oops:_oops_ioctl+48/80] [_sys_ioctl+254/272] [_system_call+82/128] 
+Aug 29 09:51:01 blizard kernel: Code: c7 00 05 00 00 00 eb 08 90 90 90 90 90 90 90 90 89 ec 5d c3 
+---------------------------------------------------------------------------
+
+Dr. G.W. Wettstein           Oncology Research Div. Computing Facility
+Roger Maris Cancer Center    INTERNET: greg@wind.rmcc.com
+820 4th St. N.
+Fargo, ND  58122
+Phone: 701-234-7556
+
+
+---------------------------------------------------------------------------
+Tainted kernels:
+
+Some oops reports contain the string 'Tainted: ' after the program
+counter, this indicates that the kernel has been tainted by some
+mechanism.  The string is followed by a series of position sensitive
+characters, each representing a particular tainted value.
+
+  1: 'G' if all modules loaded have a GPL or compatible license, 'P' if
+     any proprietary module has been loaded.  Modules without a
+     MODULE_LICENSE or with a MODULE_LICENSE that is not recognised by
+     insmod as GPL compatible are assumed to be proprietary.
+
+  2: 'F' if any module was force loaded by insmod -f, ' ' if all
+     modules were loaded normally.
+
+  3: 'S' if the oops occurred on an SMP kernel running on hardware that
+      hasn't been certified as safe to run multiprocessor.
+          Currently this occurs only on various Athlons that are not
+          SMP capable.
+
+The primary reason for the 'Tainted: ' string is to tell kernel
+debuggers if this is a clean kernel or if anything unusual has
+occurred.  Tainting is permanent, even if an offending module is
+unloading the tainted value remains to indicate that the kernel is not
+trustworthy.