1 files changed, 2536 insertions, 0 deletions

diff --git a/Documentation/s390/Debugging390.txt b/Documentation/s390/Debugging390.txt
new file mode 100644
index 000000000000..adbfe620c061
--- /dev/null
+++ b/Documentation/s390/Debugging390.txt
@@ -0,0 +1,2536 @@
+              
+                          Debugging on Linux for s/390 & z/Architecture
+                                       by
+                Denis Joseph Barrow (djbarrow@de.ibm.com,barrow_dj@yahoo.com)
+                Copyright (C) 2000-2001 IBM Deutschland Entwicklung GmbH, IBM Corporation
+                              Best viewed with fixed width fonts 
+
+Overview of Document:
+=====================
+This document is intended to give an good overview of how to debug 
+Linux for s/390 & z/Architecture it isn't intended as a complete reference & not a
+tutorial on the fundamentals of C & assembly, it dosen't go into
+390 IO in any detail. It is intended to complement the documents in the
+reference section below & any other worthwhile references you get.
+
+It is intended like the Enterprise Systems Architecture/390 Reference Summary
+to be printed out & used as a quick cheat sheet self help style reference when
+problems occur.
+
+Contents
+========
+Register Set
+Address Spaces on Intel Linux
+Address Spaces on Linux for s/390 & z/Architecture
+The Linux for s/390 & z/Architecture Kernel Task Structure
+Register Usage & Stackframes on Linux for s/390 & z/Architecture
+A sample program with comments
+Compiling programs for debugging on Linux for s/390 & z/Architecture
+Figuring out gcc compile errors
+Debugging Tools
+objdump
+strace
+Performance Debugging 
+Debugging under VM
+s/390 & z/Architecture IO Overview
+Debugging IO on s/390 & z/Architecture under VM
+GDB on s/390 & z/Architecture
+Stack chaining in gdb by hand
+Examining core dumps
+ldd
+Debugging modules
+The proc file system
+Starting points for debugging scripting languages etc.
+Dumptool & Lcrash
+SysRq
+References
+Special Thanks
+
+Register Set
+============
+The current architectures have the following registers.
+ 
+16  General propose registers, 32 bit on s/390 64 bit on z/Architecture, r0-r15 or gpr0-gpr15 used for arithmetic & addressing. 
+
+16 Control registers, 32 bit on s/390 64 bit on z/Architecture, ( cr0-cr15 kernel usage only ) used for memory management,
+interrupt control,debugging control etc.
+
+16 Access registers ( ar0-ar15 ) 32 bit on s/390 & z/Architecture
+not used by normal programs but potentially could 
+be used as temporary storage. Their main purpose is their 1 to 1
+association with general purpose registers and are used in
+the kernel for copying data between kernel & user address spaces.
+Access register 0 ( & access register 1 on z/Architecture ( needs 64 bit 
+pointer ) ) is currently used by the pthread library as a pointer to
+the current running threads private area.
+
+16 64 bit floating point registers (fp0-fp15 ) IEEE & HFP floating 
+point format compliant on G5 upwards & a Floating point control reg (FPC) 
+4  64 bit registers (fp0,fp2,fp4 & fp6) HFP only on older machines.
+Note:
+Linux (currently) always uses IEEE & emulates G5 IEEE format on older machines,
+( provided the kernel is configured for this ).
+
+
+The PSW is the most important register on the machine it
+is 64 bit on s/390 & 128 bit on z/Architecture & serves the roles of 
+a program counter (pc), condition code register,memory space designator.
+In IBM standard notation I am counting bit 0 as the MSB.
+It has several advantages over a normal program counter
+in that you can change address translation & program counter 
+in a single instruction. To change address translation,
+e.g. switching address translation off requires that you
+have a logical=physical mapping for the address you are
+currently running at.
+
+      Bit           Value
+s/390 z/Architecture
+0       0     Reserved ( must be 0 ) otherwise specification exception occurs.
+
+1       1     Program Event Recording 1 PER enabled, 
+              PER is used to facilititate debugging e.g. single stepping.
+
+2-4    2-4    Reserved ( must be 0 ). 
+
+5       5     Dynamic address translation 1=DAT on.
+
+6       6     Input/Output interrupt Mask
+
+7       7     External interrupt Mask used primarily for interprocessor signalling & 
+              clock interrupts.
+
+8-11  8-11    PSW Key used for complex memory protection mechanism not used under linux
+
+12      12    1 on s/390 0 on z/Architecture
+
+13      13    Machine Check Mask 1=enable machine check interrupts
+
+14      14    Wait State set this to 1 to stop the processor except for interrupts & give 
+              time to other LPARS used in CPU idle in the kernel to increase overall 
+              usage of processor resources.
+
+15      15    Problem state ( if set to 1 certain instructions are disabled )
+              all linux user programs run with this bit 1 
+              ( useful info for debugging under VM ).
+
+16-17 16-17   Address Space Control
+
+              00 Primary Space Mode when DAT on
+              The linux kernel currently runs in this mode, CR1 is affiliated with 
+              this mode & points to the primary segment table origin etc.
+
+              01 Access register mode this mode is used in functions to 
+              copy data between kernel & user space.
+
+              10 Secondary space mode not used in linux however CR7 the
+              register affiliated with this mode is & this & normally
+              CR13=CR7 to allow us to copy data between kernel & user space.
+              We do this as follows:
+              We set ar2 to 0 to designate its
+              affiliated gpr ( gpr2 )to point to primary=kernel space.
+              We set ar4 to 1 to designate its
+              affiliated gpr ( gpr4 ) to point to secondary=home=user space
+              & then essentially do a memcopy(gpr2,gpr4,size) to
+              copy data between the address spaces, the reason we use home space for the
+              kernel & don't keep secondary space free is that code will not run in 
+              secondary space.
+
+              11 Home Space Mode all user programs run in this mode.
+              it is affiliated with CR13.
+
+18-19 18-19   Condition codes (CC)
+
+20    20      Fixed point overflow mask if 1=FPU exceptions for this event 
+              occur ( normally 0 ) 
+
+21    21      Decimal overflow mask if 1=FPU exceptions for this event occur 
+              ( normally 0 )
+
+22    22      Exponent underflow mask if 1=FPU exceptions for this event occur 
+              ( normally 0 )
+
+23    23      Significance Mask if 1=FPU exceptions for this event occur 
+              ( normally 0 )
+
+24-31 24-30   Reserved Must be 0.
+
+      31      Extended Addressing Mode
+      32      Basic Addressing Mode
+              Used to set addressing mode
+              PSW 31   PSW 32
+                0         0        24 bit
+                0         1        31 bit
+                1         1        64 bit
+
+32             1=31 bit addressing mode 0=24 bit addressing mode (for backward 
+               compatibility ), linux always runs with this bit set to 1
+
+33-64          Instruction address.
+      33-63    Reserved must be 0
+      64-127   Address
+               In 24 bits mode bits 64-103=0 bits 104-127 Address 
+               In 31 bits mode bits 64-96=0 bits 97-127 Address
+               Note: unlike 31 bit mode on s/390 bit 96 must be zero
+               when loading the address with LPSWE otherwise a 
+               specification exception occurs, LPSW is fully backward
+               compatible.
+          
+          
+Prefix Page(s)
+--------------    
+This per cpu memory area is too intimately tied to the processor not to mention.
+It exists between the real addresses 0-4096 on s/390 & 0-8192 z/Architecture & is exchanged 
+with a 1 page on s/390 or 2 pages on z/Architecture in absolute storage by the set 
+prefix instruction in linux'es startup. 
+This page is mapped to a different prefix for each processor in an SMP configuration
+( assuming the os designer is sane of course :-) ).
+Bytes 0-512 ( 200 hex ) on s/390 & 0-512,4096-4544,4604-5119 currently on z/Architecture 
+are used by the processor itself for holding such information as exception indications & 
+entry points for exceptions.
+Bytes after 0xc00 hex are used by linux for per processor globals on s/390 & z/Architecture 
+( there is a gap on z/Architecure too currently between 0xc00 & 1000 which linux uses ).
+The closest thing to this on traditional architectures is the interrupt
+vector table. This is a good thing & does simplify some of the kernel coding
+however it means that we now cannot catch stray NULL pointers in the
+kernel without hard coded checks.
+
+
+
+Address Spaces on Intel Linux
+=============================
+
+The traditional Intel Linux is approximately mapped as follows forgive
+the ascii art.
+0xFFFFFFFF 4GB Himem                        *****************
+                                            *               *
+                                            * Kernel Space  *
+                                            *               *
+                                            *****************          ****************
+User Space Himem (typically 0xC0000000 3GB )*  User Stack   *          *              *
+                                            *****************          *              *
+                                            *  Shared Libs  *          * Next Process *          
+                                            *****************          *     to       *  
+                                            *               *    <==   *     Run      *  <==
+                                            *  User Program *          *              *
+                                            *   Data BSS    *          *              *
+                                            *    Text       *          *              *
+                                            *   Sections    *          *              *
+0x00000000                                  *****************          ****************
+
+Now it is easy to see that on Intel it is quite easy to recognise a kernel address 
+as being one greater than user space himem ( in this case 0xC0000000).
+& addresses of less than this are the ones in the current running program on this
+processor ( if an smp box ).
+If using the virtual machine ( VM ) as a debugger it is quite difficult to
+know which user process is running as the address space you are looking at
+could be from any process in the run queue.
+
+The limitation of Intels addressing technique is that the linux
+kernel uses a very simple real address to virtual addressing technique
+of Real Address=Virtual Address-User Space Himem.
+This means that on Intel the kernel linux can typically only address
+Himem=0xFFFFFFFF-0xC0000000=1GB & this is all the RAM these machines
+can typically use.
+They can lower User Himem to 2GB or lower & thus be
+able to use 2GB of RAM however this shrinks the maximum size
+of User Space from 3GB to 2GB they have a no win limit of 4GB unless
+they go to 64 Bit.
+
+
+On 390 our limitations & strengths make us slightly different.
+For backward compatibility we are only allowed use 31 bits (2GB)
+of our 32 bit addresses,however, we use entirely separate address 
+spaces for the user & kernel.
+
+This means we can support 2GB of non Extended RAM on s/390, & more
+with the Extended memory management swap device & 
+currently 4TB of physical memory currently on z/Architecture.
+
+
+Address Spaces on Linux for s/390 & z/Architecture
+==================================================
+
+Our addressing scheme is as follows
+
+
+Himem 0x7fffffff 2GB on s/390    *****************          ****************
+currently 0x3ffffffffff (2^42)-1 *  User Stack   *          *              *
+on z/Architecture.               *****************          *              *
+                                 *  Shared Libs  *          *              *      
+                                 *****************          *              *  
+                                 *               *          *    Kernel    *  
+                                 *  User Program *          *              *
+                                 *   Data BSS    *          *              *
+                                 *    Text       *          *              *
+                                 *   Sections    *          *              *
+0x00000000                       *****************          ****************
+
+This also means that we need to look at the PSW problem state bit
+or the addressing mode to decide whether we are looking at
+user or kernel space.
+
+Virtual Addresses on s/390 & z/Architecture
+===========================================
+
+A virtual address on s/390 is made up of 3 parts
+The SX ( segment index, roughly corresponding to the PGD & PMD in linux terminology ) 
+being bits 1-11.
+The PX ( page index, corresponding to the page table entry (pte) in linux terminology )
+being bits 12-19. 
+The remaining bits BX (the byte index are the offset in the page )
+i.e. bits 20 to 31.
+
+On z/Architecture in linux we currently make up an address from 4 parts.
+The region index bits (RX) 0-32 we currently use bits 22-32
+The segment index (SX) being bits 33-43
+The page index (PX) being bits  44-51
+The byte index (BX) being bits  52-63
+
+Notes:
+1) s/390 has no PMD so the PMD is really the PGD also.
+A lot of this stuff is defined in pgtable.h.
+
+2) Also seeing as s/390's page indexes are only 1k  in size 
+(bits 12-19 x 4 bytes per pte ) we use 1 ( page 4k )
+to make the best use of memory by updating 4 segment indices 
+entries each time we mess with a PMD & use offsets 
+0,1024,2048 & 3072 in this page as for our segment indexes.
+On z/Architecture our page indexes are now 2k in size
+( bits 12-19 x 8 bytes per pte ) we do a similar trick
+but only mess with 2 segment indices each time we mess with
+a PMD.
+
+3) As z/Architecture supports upto a massive 5-level page table lookup we 
+can only use 3 currently on Linux ( as this is all the generic kernel
+currently supports ) however this may change in future
+this allows us to access ( according to my sums )
+4TB of virtual storage per process i.e.
+4096*512(PTES)*1024(PMDS)*2048(PGD) = 4398046511104 bytes,
+enough for another 2 or 3 of years I think :-).
+to do this we use a region-third-table designation type in
+our address space control registers.
+ 
+
+The Linux for s/390 & z/Architecture Kernel Task Structure
+==========================================================
+Each process/thread under Linux for S390 has its own kernel task_struct
+defined in linux/include/linux/sched.h
+The S390 on initialisation & resuming of a process on a cpu sets
+the __LC_KERNEL_STACK variable in the spare prefix area for this cpu
+( which we use for per processor globals).
+
+The kernel stack pointer is intimately tied with the task stucture for
+each processor as follows.
+
+                      s/390
+            ************************
+            *  1 page kernel stack *
+            *        ( 4K )        *
+            ************************
+            *   1 page task_struct *        
+            *        ( 4K )        *
+8K aligned  ************************ 
+
+                 z/Architecture
+            ************************
+            *  2 page kernel stack *
+            *        ( 8K )        *
+            ************************
+            *  2 page task_struct  *        
+            *        ( 8K )        *
+16K aligned ************************ 
+
+What this means is that we don't need to dedicate any register or global variable
+to point to the current running process & can retrieve it with the following
+very simple construct for s/390 & one very similar for z/Architecture.
+
+static inline struct task_struct * get_current(void)
+{
+        struct task_struct *current;
+        __asm__("lhi   %0,-8192\n\t"
+                "nr    %0,15"
+                : "=r" (current) );
+        return current;
+}
+
+i.e. just anding the current kernel stack pointer with the mask -8192.
+Thankfully because Linux dosen't have support for nested IO interrupts
+& our devices have large buffers can survive interrupts being shut for 
+short amounts of time we don't need a separate stack for interrupts.
+
+
+
+
+Register Usage & Stackframes on Linux for s/390 & z/Architecture
+=================================================================
+Overview:
+---------
+This is the code that gcc produces at the top & the bottom of
+each function, it usually is fairly consistent & similar from 
+function to function & if you know its layout you can probalby
+make some headway in finding the ultimate cause of a problem
+after a crash without a source level debugger.
+
+Note: To follow stackframes requires a knowledge of C or Pascal &
+limited knowledge of one assembly language.
+
+It should be noted that there are some differences between the
+s/390 & z/Architecture stack layouts as the z/Architecture stack layout didn't have
+to maintain compatibility with older linkage formats.
+
+Glossary:
+---------
+alloca:
+This is a built in compiler function for runtime allocation
+of extra space on the callers stack which is obviously freed
+up on function exit ( e.g. the caller may choose to allocate nothing
+of a buffer of 4k if required for temporary purposes ), it generates 
+very efficient code ( a few cycles  ) when compared to alternatives 
+like malloc.
+
+automatics: These are local variables on the stack,
+i.e they aren't in registers & they aren't static.
+
+back-chain:
+This is a pointer to the stack pointer before entering a
+framed functions ( see frameless function ) prologue got by 
+deferencing the address of the current stack pointer,
+ i.e. got by accessing the 32 bit value at the stack pointers
+current location.
+
+base-pointer:
+This is a pointer to the back of the literal pool which
+is an area just behind each procedure used to store constants
+in each function.
+
+call-clobbered: The caller probably needs to save these registers if there 
+is something of value in them, on the stack or elsewhere before making a 
+call to another procedure so that it can restore it later.
+
+epilogue:
+The code generated by the compiler to return to the caller.
+
+frameless-function
+A frameless function in Linux for s390 & z/Architecture is one which doesn't 
+need more than the register save area ( 96 bytes on s/390, 160 on z/Architecture )
+given to it by the caller.
+A frameless function never:
+1) Sets up a back chain.
+2) Calls alloca.
+3) Calls other normal functions
+4) Has automatics.
+
+GOT-pointer:
+This is a pointer to the global-offset-table in ELF
+( Executable Linkable Format, Linux'es most common executable format ),
+all globals & shared library objects are found using this pointer.
+
+lazy-binding
+ELF shared libraries are typically only loaded when routines in the shared
+library are actually first called at runtime. This is lazy binding.
+
+procedure-linkage-table
+This is a table found from the GOT which contains pointers to routines
+in other shared libraries which can't be called to by easier means.
+
+prologue:
+The code generated by the compiler to set up the stack frame.
+
+outgoing-args:
+This is extra area allocated on the stack of the calling function if the
+parameters for the callee's cannot all be put in registers, the same
+area can be reused by each function the caller calls.
+
+routine-descriptor:
+A COFF  executable format based concept of a procedure reference 
+actually being 8 bytes or more as opposed to a simple pointer to the routine.
+This is typically defined as follows
+Routine Descriptor offset 0=Pointer to Function
+Routine Descriptor offset 4=Pointer to Table of Contents
+The table of contents/TOC is roughly equivalent to a GOT pointer.
+& it means that shared libraries etc. can be shared between several
+environments each with their own TOC.
+
+ 
+static-chain: This is used in nested functions a concept adopted from pascal 
+by gcc not used in ansi C or C++ ( although quite useful ), basically it
+is a pointer used to reference local variables of enclosing functions.
+You might come across this stuff once or twice in your lifetime.
+
+e.g.
+The function below should return 11 though gcc may get upset & toss warnings 
+about unused variables.
+int FunctionA(int a)
+{
+        int b;
+        FunctionC(int c)
+        {
+                b=c+1;
+        }
+        FunctionC(10);
+        return(b);
+}
+
+
+s/390 & z/Architecture Register usage
+=====================================
+r0       used by syscalls/assembly                  call-clobbered
+r1       used by syscalls/assembly                  call-clobbered
+r2       argument 0 / return value 0                call-clobbered
+r3       argument 1 / return value 1 (if long long) call-clobbered
+r4       argument 2                                 call-clobbered
+r5       argument 3                                 call-clobbered
+r6       argument 5                                 saved
+r7       pointer-to arguments 5 to ...              saved      
+r8       this & that                                saved
+r9       this & that                                saved
+r10      static-chain ( if nested function )        saved
+r11      frame-pointer ( if function used alloca )  saved
+r12      got-pointer                                saved
+r13      base-pointer                               saved
+r14      return-address                             saved
+r15      stack-pointer                              saved
+
+f0       argument 0 / return value ( float/double ) call-clobbered
+f2       argument 1                                 call-clobbered
+f4       z/Architecture argument 2                  saved
+f6       z/Architecture argument 3                  saved
+The remaining floating points
+f1,f3,f5 f7-f15 are call-clobbered.
+
+Notes:
+------
+1) The only requirement is that registers which are used
+by the callee are saved, e.g. the compiler is perfectly
+capible of using r11 for purposes other than a frame a
+frame pointer if a frame pointer is not needed.
+2) In functions with variable arguments e.g. printf the calling procedure 
+is identical to one without variable arguments & the same number of 
+parameters. However, the prologue of this function is somewhat more
+hairy owing to it having to move these parameters to the stack to
+get va_start, va_arg & va_end to work.
+3) Access registers are currently unused by gcc but are used in
+the kernel. Possibilities exist to use them at the moment for
+temporary storage but it isn't recommended.
+4) Only 4 of the floating point registers are used for
+parameter passing as older machines such as G3 only have only 4
+& it keeps the stack frame compatible with other compilers.
+However with IEEE floating point emulation under linux on the
+older machines you are free to use the other 12.
+5) A long long or double parameter cannot be have the 
+first 4 bytes in a register & the second four bytes in the 
+outgoing args area. It must be purely in the outgoing args
+area if crossing this boundary.
+6) Floating point parameters are mixed with outgoing args
+on the outgoing args area in the order the are passed in as parameters.
+7) Floating point arguments 2 & 3 are saved in the outgoing args area for 
+z/Architecture
+
+
+Stack Frame Layout
+------------------
+s/390     z/Architecture
+0         0             back chain ( a 0 here signifies end of back chain )
+4         8             eos ( end of stack, not used on Linux for S390 used in other linkage formats )
+8         16            glue used in other s/390 linkage formats for saved routine descriptors etc.
+12        24            glue used in other s/390 linkage formats for saved routine descriptors etc.
+16        32            scratch area
+20        40            scratch area
+24        48            saved r6 of caller function
+28        56            saved r7 of caller function
+32        64            saved r8 of caller function
+36        72            saved r9 of caller function
+40        80            saved r10 of caller function
+44        88            saved r11 of caller function
+48        96            saved r12 of caller function
+52        104           saved r13 of caller function
+56        112           saved r14 of caller function
+60        120           saved r15 of caller function
+64        128           saved f4 of caller function
+72        132           saved f6 of caller function
+80                      undefined
+96        160           outgoing args passed from caller to callee
+96+x      160+x         possible stack alignment ( 8 bytes desirable )
+96+x+y    160+x+y       alloca space of caller ( if used )
+96+x+y+z  160+x+y+z     automatics of caller ( if used )
+0                       back-chain
+
+A sample program with comments.
+===============================
+
+Comments on the function test
+-----------------------------
+1) It didn't need to set up a pointer to the constant pool gpr13 as it isn't used
+( :-( ).
+2) This is a frameless function & no stack is bought.
+3) The compiler was clever enough to recognise that it could return the
+value in r2 as well as use it for the passed in parameter ( :-) ).
+4) The basr ( branch relative & save ) trick works as follows the instruction 
+has a special case with r0,r0 with some instruction operands is understood as 
+the literal value 0, some risc architectures also do this ). So now
+we are branching to the next address & the address new program counter is
+in r13,so now we subtract the size of the function prologue we have executed
++ the size of the literal pool to get to the top of the literal pool
+0040037c int test(int b)
+{                                                          # Function prologue below
+  40037c:       90 de f0 34     stm     %r13,%r14,52(%r15) # Save registers r13 & r14
+  400380:       0d d0           basr    %r13,%r0           # Set up pointer to constant pool using
+  400382:       a7 da ff fa     ahi     %r13,-6            # basr trick
+        return(5+b);
+                                                           # Huge main program
+  400386:       a7 2a 00 05     ahi     %r2,5              # add 5 to r2
+
+                                                           # Function epilogue below 
+  40038a:       98 de f0 34     lm      %r13,%r14,52(%r15) # restore registers r13 & 14
+  40038e:       07 fe           br      %r14               # return
+}
+
+Comments on the function main
+-----------------------------
+1) The compiler did this function optimally ( 8-) )
+
+Literal pool for main.
+400390: ff ff ff ec     .long 0xffffffec
+main(int argc,char *argv[])
+{                                                          # Function prologue below
+  400394:       90 bf f0 2c     stm     %r11,%r15,44(%r15) # Save necessary registers
+  400398:       18 0f           lr      %r0,%r15           # copy stack pointer to r0
+  40039a:       a7 fa ff a0     ahi     %r15,-96           # Make area for callee saving 
+  40039e:       0d d0           basr    %r13,%r0           # Set up r13 to point to
+  4003a0:       a7 da ff f0     ahi     %r13,-16           # literal pool
+  4003a4:       50 00 f0 00     st      %r0,0(%r15)        # Save backchain
+
+        return(test(5));                                   # Main Program Below
+  4003a8:       58 e0 d0 00     l       %r14,0(%r13)       # load relative address of test from
+                                                           # literal pool
+  4003ac:       a7 28 00 05     lhi     %r2,5              # Set first parameter to 5
+  4003b0:       4d ee d0 00     bas     %r14,0(%r14,%r13)  # jump to test setting r14 as return
+                                                           # address using branch & save instruction.
+
+                                                           # Function Epilogue below
+  4003b4:       98 bf f0 8c     lm      %r11,%r15,140(%r15)# Restore necessary registers.
+  4003b8:       07 fe           br      %r14               # return to do program exit 
+}
+
+
+Compiler updates
+----------------
+
+main(int argc,char *argv[])
+{
+  4004fc:       90 7f f0 1c             stm     %r7,%r15,28(%r15)
+  400500:       a7 d5 00 04             bras    %r13,400508 <main+0xc>
+  400504:       00 40 04 f4             .long   0x004004f4 
+  # compiler now puts constant pool in code to so it saves an instruction 
+  400508:       18 0f                   lr      %r0,%r15
+  40050a:       a7 fa ff a0             ahi     %r15,-96
+  40050e:       50 00 f0 00             st      %r0,0(%r15)
+        return(test(5));
+  400512:       58 10 d0 00             l       %r1,0(%r13)
+  400516:       a7 28 00 05             lhi     %r2,5
+  40051a:       0d e1                   basr    %r14,%r1
+  # compiler adds 1 extra instruction to epilogue this is done to
+  # avoid processor pipeline stalls owing to data dependencies on g5 &
+  # above as register 14 in the old code was needed directly after being loaded 
+  # by the lm   %r11,%r15,140(%r15) for the br %14.
+  40051c:       58 40 f0 98             l       %r4,152(%r15)
+  400520:       98 7f f0 7c             lm      %r7,%r15,124(%r15)
+  400524:       07 f4                   br      %r4
+}
+
+
+Hartmut ( our compiler developer ) also has been threatening to take out the
+stack backchain in optimised code as this also causes pipeline stalls, you
+have been warned.
+
+64 bit z/Architecture code disassembly
+--------------------------------------
+
+If you understand the stuff above you'll understand the stuff
+below too so I'll avoid repeating myself & just say that 
+some of the instructions have g's on the end of them to indicate
+they are 64 bit & the stack offsets are a bigger, 
+the only other difference you'll find between 32 & 64 bit is that
+we now use f4 & f6 for floating point arguments on 64 bit.
+00000000800005b0 <test>:
+int test(int b)
+{
+        return(5+b);
+    800005b0:   a7 2a 00 05             ahi     %r2,5
+    800005b4:   b9 14 00 22             lgfr    %r2,%r2 # downcast to integer
+    800005b8:   07 fe                   br      %r14
+    800005ba:   07 07                   bcr     0,%r7
+
+
+}
+
+00000000800005bc <main>:
+main(int argc,char *argv[])
+{ 
+    800005bc:   eb bf f0 58 00 24       stmg    %r11,%r15,88(%r15)
+    800005c2:   b9 04 00 1f             lgr     %r1,%r15
+    800005c6:   a7 fb ff 60             aghi    %r15,-160
+    800005ca:   e3 10 f0 00 00 24       stg     %r1,0(%r15)
+        return(test(5));
+    800005d0:   a7 29 00 05             lghi    %r2,5
+    # brasl allows jumps > 64k & is overkill here bras would do fune
+    800005d4:   c0 e5 ff ff ff ee       brasl   %r14,800005b0 <test> 
+    800005da:   e3 40 f1 10 00 04       lg      %r4,272(%r15)
+    800005e0:   eb bf f0 f8 00 04       lmg     %r11,%r15,248(%r15)
+    800005e6:   07 f4                   br      %r4
+}
+
+
+
+Compiling programs for debugging on Linux for s/390 & z/Architecture
+====================================================================
+-gdwarf-2 now works it should be considered the default debugging
+format for s/390 & z/Architecture as it is more reliable for debugging
+shared libraries,  normal -g debugging works much better now
+Thanks to the IBM java compiler developers bug reports. 
+
+This is typically done adding/appending the flags -g or -gdwarf-2 to the 
+CFLAGS & LDFLAGS variables Makefile of the program concerned.
+
+If using gdb & you would like accurate displays of registers &
+ stack traces compile without optimisation i.e make sure
+that there is no -O2 or similar on the CFLAGS line of the Makefile &
+the emitted gcc commands, obviously this will produce worse code 
+( not advisable for shipment ) but it is an  aid to the debugging process.
+
+This aids debugging because the compiler will copy parameters passed in
+in registers onto the stack so backtracing & looking at passed in
+parameters will work, however some larger programs which use inline functions
+will not compile without optimisation.
+
+Debugging with optimisation has since much improved after fixing
+some bugs, please make sure you are using gdb-5.0 or later developed 
+after Nov'2000.
+
+Figuring out gcc compile errors
+===============================
+If you are getting a lot of syntax errors compiling a program & the problem
+isn't blatantly obvious from the source.
+It often helps to just preprocess the file, this is done with the -E
+option in gcc.
+What this does is that it runs through the very first phase of compilation
+( compilation in gcc is done in several stages & gcc calls many programs to
+achieve its end result ) with the -E option gcc just calls the gcc preprocessor (cpp).
+The c preprocessor does the following, it joins all the files #included together
+recursively ( #include files can #include other files ) & also the c file you wish to compile.
+It puts a fully qualified path of the #included files in a comment & it
+does macro expansion.
+This is useful for debugging because
+1) You can double check whether the files you expect to be included are the ones
+that are being included ( e.g. double check that you aren't going to the i386 asm directory ).
+2) Check that macro definitions aren't clashing with typedefs,
+3) Check that definitons aren't being used before they are being included.
+4) Helps put the line emitting the error under the microscope if it contains macros.
+
+For convenience the Linux kernel's makefile will do preprocessing automatically for you
+by suffixing the file you want built with .i ( instead of .o )
+
+e.g.
+from the linux directory type
+make arch/s390/kernel/signal.i
+this will build
+
+s390-gcc -D__KERNEL__ -I/home1/barrow/linux/include -Wall -Wstrict-prototypes -O2 -fomit-frame-pointer
+-fno-strict-aliasing -D__SMP__ -pipe -fno-strength-reduce   -E arch/s390/kernel/signal.c
+> arch/s390/kernel/signal.i  
+
+Now look at signal.i you should see something like.
+
+
+# 1 "/home1/barrow/linux/include/asm/types.h" 1
+typedef unsigned short umode_t;
+typedef __signed__ char __s8;
+typedef unsigned char __u8;
+typedef __signed__ short __s16;
+typedef unsigned short __u16;
+
+If instead you are getting errors further down e.g.
+unknown instruction:2515 "move.l" or better still unknown instruction:2515 
+"Fixme not implemented yet, call Martin" you are probably are attempting to compile some code 
+meant for another architecture or code that is simply not implemented, with a fixme statement
+stuck into the inline assembly code so that the author of the file now knows he has work to do.
+To look at the assembly emitted by gcc just before it is about to call gas ( the gnu assembler )
+use the -S option.
+Again for your convenience the Linux kernel's Makefile will hold your hand &
+do all this donkey work for you also by building the file with the .s suffix.
+e.g.
+from the Linux directory type 
+make arch/s390/kernel/signal.s 
+
+s390-gcc -D__KERNEL__ -I/home1/barrow/linux/include -Wall -Wstrict-prototypes -O2 -fomit-frame-pointer
+-fno-strict-aliasing -D__SMP__ -pipe -fno-strength-reduce  -S arch/s390/kernel/signal.c 
+-o arch/s390/kernel/signal.s  
+
+
+This will output something like, ( please note the constant pool & the useful comments
+in the prologue to give you a hand at interpreting it ).
+
+.LC54:
+        .string "misaligned (__u16 *) in __xchg\n"
+.LC57:
+        .string "misaligned (__u32 *) in __xchg\n"
+.L$PG1: # Pool sys_sigsuspend
+.LC192:
+        .long   -262401
+.LC193:
+        .long   -1
+.LC194:
+        .long   schedule-.L$PG1
+.LC195:
+        .long   do_signal-.L$PG1
+        .align 4
+.globl sys_sigsuspend
+        .type    sys_sigsuspend,@function
+sys_sigsuspend:
+#       leaf function           0
+#       automatics              16
+#       outgoing args           0
+#       need frame pointer      0
+#       call alloca             0
+#       has varargs             0
+#       incoming args (stack)   0
+#       function length         168
+        STM     8,15,32(15)
+        LR      0,15
+        AHI     15,-112
+        BASR    13,0
+.L$CO1: AHI     13,.L$PG1-.L$CO1
+        ST      0,0(15)
+        LR    8,2
+        N     5,.LC192-.L$PG1(13) 
+
+Adding -g to the above output makes the output even more useful
+e.g. typing
+make CC:="s390-gcc -g" kernel/sched.s
+
+which compiles.
+s390-gcc -g -D__KERNEL__ -I/home/barrow/linux-2.3/include -Wall -Wstrict-prototypes -O2 -fomit-frame-pointer -fno-strict-aliasing -pipe -fno-strength-reduce   -S kernel/sched.c -o kernel/sched.s 
+
+also outputs stabs ( debugger ) info, from this info you can find out the
+offsets & sizes of various elements in structures.
+e.g. the stab for the structure
+struct rlimit {
+        unsigned long   rlim_cur;
+        unsigned long   rlim_max;
+};
+is
+.stabs "rlimit:T(151,2)=s8rlim_cur:(0,5),0,32;rlim_max:(0,5),32,32;;",128,0,0,0
+from this stab you can see that 
+rlimit_cur starts at bit offset 0 & is 32 bits in size
+rlimit_max starts at bit offset 32 & is 32 bits in size.
+
+
+Debugging Tools:
+================
+
+objdump
+=======
+This is a tool with many options the most useful being ( if compiled with -g).
+objdump --source <victim program or object file> > <victims debug listing >
+
+
+The whole kernel can be compiled like this ( Doing this will make a 17MB kernel
+& a 200 MB listing ) however you have to strip it before building the image
+using the strip command to make it a more reasonable size to boot it.
+
+A source/assembly mixed dump of the kernel can be done with the line
+objdump --source vmlinux > vmlinux.lst
+Also if the file isn't compiled -g this will output as much debugging information
+as it can ( e.g. function names ), however, this is very slow as it spends lots
+of time searching for debugging info, the following self explanitory line should be used 
+instead if the code isn't compiled -g.
+objdump --disassemble-all --syms vmlinux > vmlinux.lst  
+as it is much faster
+
+As hard drive space is valuble most of us use the following approach.
+1) Look at the emitted psw on the console to find the crash address in the kernel.
+2) Look at the file System.map ( in the linux directory ) produced when building 
+the kernel to find the closest address less than the current PSW to find the
+offending function.
+3) use grep or similar to search the source tree looking for the source file
+ with this function if you don't know where it is.
+4) rebuild this object file with -g on, as an example suppose the file was
+( /arch/s390/kernel/signal.o ) 
+5) Assuming the file with the erroneous function is signal.c Move to the base of the 
+Linux source tree.
+6) rm /arch/s390/kernel/signal.o
+7) make /arch/s390/kernel/signal.o
+8) watch the gcc command line emitted
+9) type it in again or alernatively cut & paste it on the console adding the -g option.
+10) objdump --source arch/s390/kernel/signal.o > signal.lst
+This will output the source & the assembly intermixed, as the snippet below shows
+This will unfortunately output addresses which aren't the same
+as the kernel ones you should be able to get around the mental arithmetic
+by playing with the --adjust-vma parameter to objdump.
+
+
+
+
+extern inline void spin_lock(spinlock_t *lp)
+{
+      a0:       18 34           lr      %r3,%r4
+      a2:       a7 3a 03 bc     ahi     %r3,956
+        __asm__ __volatile("    lhi   1,-1\n"
+      a6:       a7 18 ff ff     lhi     %r1,-1
+      aa:       1f 00           slr     %r0,%r0
+      ac:       ba 01 30 00     cs      %r0,%r1,0(%r3)
+      b0:       a7 44 ff fd     jm      aa <sys_sigsuspend+0x2e>
+        saveset = current->blocked;
+      b4:       d2 07 f0 68     mvc     104(8,%r15),972(%r4)
+      b8:       43 cc
+        return (set->sig[0] & mask) != 0;
+} 
+
+6) If debugging under VM go down to that section in the document for more info.
+
+
+I now have a tool which takes the pain out of --adjust-vma
+& you are able to do something like
+make /arch/s390/kernel/traps.lst
+& it automatically generates the correctly relocated entries for
+the text segment in traps.lst.
+This tool is now standard in linux distro's in scripts/makelst
+
+strace:
+-------
+Q. What is it ?
+A. It is a tool for intercepting calls to the kernel & logging them
+to a file & on the screen.
+
+Q. What use is it ?
+A. You can used it to find out what files a particular program opens.
+
+
+
+Example 1
+---------
+If you wanted to know does ping work but didn't have the source 
+strace ping -c 1 127.0.0.1  
+& then look at the man pages for each of the syscalls below,
+( In fact this is sometimes easier than looking at some spagetti
+source which conditionally compiles for several architectures )
+Not everything that it throws out needs to make sense immeadiately
+
+Just looking quickly you can see that it is making up a RAW socket
+for the ICMP protocol.
+Doing an alarm(10) for a 10 second timeout
+& doing a gettimeofday call before & after each read to see 
+how long the replies took, & writing some text to stdout so the user
+has an idea what is going on.
+
+socket(PF_INET, SOCK_RAW, IPPROTO_ICMP) = 3
+getuid()                                = 0
+setuid(0)                               = 0
+stat("/usr/share/locale/C/libc.cat", 0xbffff134) = -1 ENOENT (No such file or directory)
+stat("/usr/share/locale/libc/C", 0xbffff134) = -1 ENOENT (No such file or directory)
+stat("/usr/local/share/locale/C/libc.cat", 0xbffff134) = -1 ENOENT (No such file or directory)
+getpid()                                = 353
+setsockopt(3, SOL_SOCKET, SO_BROADCAST, [1], 4) = 0
+setsockopt(3, SOL_SOCKET, SO_RCVBUF, [49152], 4) = 0
+fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(3, 1), ...}) = 0
+mmap(0, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40008000
+ioctl(1, TCGETS, {B9600 opost isig icanon echo ...}) = 0
+write(1, "PING 127.0.0.1 (127.0.0.1): 56 d"..., 42PING 127.0.0.1 (127.0.0.1): 56 data bytes
+) = 42
+sigaction(SIGINT, {0x8049ba0, [], SA_RESTART}, {SIG_DFL}) = 0 
+sigaction(SIGALRM, {0x8049600, [], SA_RESTART}, {SIG_DFL}) = 0
+gettimeofday({948904719, 138951}, NULL) = 0
+sendto(3, "\10\0D\201a\1\0\0\17#\2178\307\36"..., 64, 0, {sin_family=AF_INET,
+sin_port=htons(0), sin_addr=inet_addr("127.0.0.1")}, 16) = 64
+sigaction(SIGALRM, {0x8049600, [], SA_RESTART}, {0x8049600, [], SA_RESTART}) = 0
+sigaction(SIGALRM, {0x8049ba0, [], SA_RESTART}, {0x8049600, [], SA_RESTART}) = 0
+alarm(10)                               = 0
+recvfrom(3, "E\0\0T\0005\0\0@\1|r\177\0\0\1\177"..., 192, 0, 
+{sin_family=AF_INET, sin_port=htons(50882), sin_addr=inet_addr("127.0.0.1")}, [16]) = 84
+gettimeofday({948904719, 160224}, NULL) = 0
+recvfrom(3, "E\0\0T\0006\0\0\377\1\275p\177\0"..., 192, 0, 
+{sin_family=AF_INET, sin_port=htons(50882), sin_addr=inet_addr("127.0.0.1")}, [16]) = 84
+gettimeofday({948904719, 166952}, NULL) = 0
+write(1, "64 bytes from 127.0.0.1: icmp_se"..., 
+5764 bytes from 127.0.0.1: icmp_seq=0 ttl=255 time=28.0 ms
+
+Example 2
+---------
+strace passwd 2>&1 | grep open
+produces the following output
+open("/etc/ld.so.cache", O_RDONLY)      = 3
+open("/opt/kde/lib/libc.so.5", O_RDONLY) = -1 ENOENT (No such file or directory)
+open("/lib/libc.so.5", O_RDONLY)        = 3
+open("/dev", O_RDONLY)                  = 3
+open("/var/run/utmp", O_RDONLY)         = 3
+open("/etc/passwd", O_RDONLY)           = 3
+open("/etc/shadow", O_RDONLY)           = 3
+open("/etc/login.defs", O_RDONLY)       = 4
+open("/dev/tty", O_RDONLY)              = 4 
+
+The 2>&1 is done to redirect stderr to stdout & grep is then filtering this input 
+through the pipe for each line containing the string open.
+
+
+Example 3
+---------
+Getting sophistocated
+telnetd crashes on & I don't know why
+Steps
+-----
+1) Replace the following line in /etc/inetd.conf
+telnet  stream  tcp     nowait  root    /usr/sbin/in.telnetd -h 
+with
+telnet  stream  tcp     nowait  root    /blah
+
+2) Create the file /blah with the following contents to start tracing telnetd 
+#!/bin/bash
+/usr/bin/strace -o/t1 -f /usr/sbin/in.telnetd -h 
+3) chmod 700 /blah to make it executable only to root
+4)
+killall -HUP inetd
+or ps aux | grep inetd
+get inetd's process id
+& kill -HUP inetd to restart it.
+
+Important options
+-----------------
+-o is used to tell strace to output to a file in our case t1 in the root directory
+-f is to follow children i.e.
+e.g in our case above telnetd will start the login process & subsequently a shell like bash.
+You will be able to tell which is which from the process ID's listed on the left hand side
+of the strace output.
+-p<pid> will tell strace to attach to a running process, yup this can be done provided
+ it isn't being traced or debugged already & you have enough privileges,
+the reason 2 processes cannot trace or debug the same program is that strace
+becomes the parent process of the one being debugged & processes ( unlike people )
+can have only one parent.
+
+
+However the file /t1 will get big quite quickly
+to test it telnet 127.0.0.1
+
+now look at what files in.telnetd execve'd
+413   execve("/usr/sbin/in.telnetd", ["/usr/sbin/in.telnetd", "-h"], [/* 17 vars */]) = 0
+414   execve("/bin/login", ["/bin/login", "-h", "localhost", "-p"], [/* 2 vars */]) = 0 
+
+Whey it worked!.
+
+
+Other hints:
+------------
+If the program is not very interactive ( i.e. not much keyboard input )
+& is crashing in one architecture but not in another you can do 
+an strace of both programs under as identical a scenario as you can
+on both architectures outputting to a file then.
+do a diff of the two traces using the diff program
+i.e.
+diff output1 output2
+& maybe you'll be able to see where the call paths differed, this
+is possibly near the cause of the crash. 
+
+More info
+---------
+Look at man pages for strace & the various syscalls
+e.g. man strace, man alarm, man socket.
+
+
+Performance Debugging
+=====================
+gcc is capible of compiling in profiling code just add the -p option
+to the CFLAGS, this obviously affects program size & performance.
+This can be used by the gprof gnu profiling tool or the
+gcov the gnu code coverage tool ( code coverage is a means of testing
+code quality by checking if all the code in an executable in exercised by
+a tester ).
+
+
+Using top to find out where processes are sleeping in the kernel
+----------------------------------------------------------------
+To do this copy the System.map from the root directory where
+the linux kernel was built to the /boot directory on your 
+linux machine.
+Start top
+Now type fU<return>
+You should see a new field called WCHAN which
+tells you where each process is sleeping here is a typical output.
+ 
+ 6:59pm  up 41 min,  1 user,  load average: 0.00, 0.00, 0.00
+28 processes: 27 sleeping, 1 running, 0 zombie, 0 stopped
+CPU states:  0.0% user,  0.1% system,  0.0% nice, 99.8% idle
+Mem:   254900K av,   45976K used,  208924K free,       0K shrd,   28636K buff
+Swap:       0K av,       0K used,       0K free                    8620K cached
+
+  PID USER     PRI  NI  SIZE  RSS SHARE WCHAN     STAT  LIB %CPU %MEM   TIME COMMAND
+  750 root      12   0   848  848   700 do_select S       0  0.1  0.3   0:00 in.telnetd
+  767 root      16   0  1140 1140   964           R       0  0.1  0.4   0:00 top
+    1 root       8   0   212  212   180 do_select S       0  0.0  0.0   0:00 init
+    2 root       9   0     0    0     0 down_inte SW      0  0.0  0.0   0:00 kmcheck
+
+The time command
+----------------
+Another related command is the time command which gives you an indication
+of where a process is spending the majority of its time.
+e.g.
+time ping -c 5 nc
+outputs
+real    0m4.054s
+user    0m0.010s
+sys     0m0.010s
+
+Debugging under VM
+==================
+
+Notes
+-----
+Addresses & values in the VM debugger are always hex never decimal
+Address ranges are of the format <HexValue1>-<HexValue2> or <HexValue1>.<HexValue2> 
+e.g. The address range  0x2000 to 0x3000 can be described described as
+2000-3000 or 2000.1000
+
+The VM Debugger is case insensitive.
+
+VM's strengths are usually other debuggers weaknesses you can get at any resource
+no matter how sensitive e.g. memory management resources,change address translation
+in the PSW. For kernel hacking you will reap dividends if you get good at it.
+
+The VM Debugger displays operators but not operands, probably because some
+of it was written when memory was expensive & the programmer was probably proud that
+it fitted into 2k of memory & the programmers & didn't want to shock hardcore VM'ers by
+changing the interface :-), also the debugger displays useful information on the same line & 
+the author of the code probably felt that it was a good idea not to go over 
+the 80 columns on the screen. 
+
+As some of you are probably in a panic now this isn't as unintuitive as it may seem
+as the 390 instructions are easy to decode mentally & you can make a good guess at a lot 
+of them as all the operands are nibble ( half byte aligned ) & if you have an objdump listing
+also it is quite easy to follow, if you don't have an objdump listing keep a copy of
+the s/390 Reference Summary & look at between pages 2 & 7 or alternatively the
+s/390 principles of operation.
+e.g. even I can guess that 
+0001AFF8' LR    180F        CC 0
+is a ( load register ) lr r0,r15 
+
+Also it is very easy to tell the length of a 390 instruction from the 2 most significant
+bits in the instruction ( not that this info is really useful except if you are trying to
+make sense of a hexdump of code ).
+Here is a table
+Bits                    Instruction Length
+------------------------------------------
+00                          2 Bytes
+01                          4 Bytes
+10                          4 Bytes
+11                          6 Bytes
+
+
+
+
+The debugger also displays other useful info on the same line such as the
+addresses being operated on destination addresses of branches & condition codes.
+e.g.  
+00019736' AHI   A7DAFF0E    CC 1
+000198BA' BRC   A7840004 -> 000198C2'   CC 0
+000198CE' STM   900EF068 >> 0FA95E78    CC 2
+
+
+
+Useful VM debugger commands
+---------------------------
+
+I suppose I'd better mention this before I start
+to list the current active traces do 
+Q TR
+there can be a maximum of 255 of these per set
+( more about trace sets later ).
+To stop traces issue a
+TR END.
+To delete a particular breakpoint issue
+TR DEL <breakpoint number>
+
+The PA1 key drops to CP mode so you can issue debugger commands,
+Doing alt c (on my 3270 console at least ) clears the screen. 
+hitting b <enter> comes back to the running operating system
+from cp mode ( in our case linux ).
+It is typically useful to add shortcuts to your profile.exec file
+if you have one ( this is roughly equivalent to autoexec.bat in DOS ).
+file here are a few from mine.
+/* this gives me command history on issuing f12 */
+set pf12 retrieve 
+/* this continues */
+set pf8 imm b
+/* goes to trace set a */
+set pf1 imm tr goto a
+/* goes to trace set b */
+set pf2 imm tr goto b
+/* goes to trace set c */
+set pf3 imm tr goto c
+
+
+
+Instruction Tracing
+-------------------
+Setting a simple breakpoint
+TR I PSWA <address>
+To debug a particular function try
+TR I R <function address range>
+TR I on its own will single step.
+TR I DATA <MNEMONIC> <OPTIONAL RANGE> will trace for particular mnemonics
+e.g.
+TR I DATA 4D R 0197BC.4000
+will trace for BAS'es ( opcode 4D ) in the range 0197BC.4000
+if you were inclined you could add traces for all branch instructions &
+suffix them with the run prefix so you would have a backtrace on screen 
+when a program crashes.
+TR BR <INTO OR FROM> will trace branches into or out of an address.
+e.g.
+TR BR INTO 0 is often quite useful if a program is getting awkward & deciding
+to branch to 0 & crashing as this will stop at the address before in jumps to 0.
+TR I R <address range> RUN cmd d g
+single steps a range of addresses but stays running &
+displays the gprs on each step.
+
+
+
+Displaying & modifying Registers
+--------------------------------
+D G will display all the gprs
+Adding a extra G to all the commands is necessary to access the full 64 bit 
+content in VM on z/Architecture obviously this isn't required for access registers
+as these are still 32 bit.
+e.g. DGG instead of DG 
+D X will display all the control registers
+D AR will display all the access registers
+D AR4-7 will display access registers 4 to 7
+CPU ALL D G will display the GRPS of all CPUS in the configuration
+D PSW will display the current PSW
+st PSW 2000 will put the value 2000 into the PSW &
+cause crash your machine.
+D PREFIX displays the prefix offset
+
+
+Displaying Memory
+-----------------
+To display memory mapped using the current PSW's mapping try
+D <range>
+To make VM display a message each time it hits a particular address & continue try
+D I<range> will disassemble/display a range of instructions.
+ST addr 32 bit word will store a 32 bit aligned address
+D T<range> will display the EBCDIC in an address ( if you are that way inclined )
+D R<range> will display real addresses ( without DAT ) but with prefixing.
+There are other complex options to display if you need to get at say home space
+but are in primary space the easiest thing to do is to temporarily
+modify the PSW to the other addressing mode, display the stuff & then
+restore it.
+
+
+ 
+Hints
+-----
+If you want to issue a debugger command without halting your virtual machine with the
+PA1 key try prefixing the command with #CP e.g.
+#cp tr i pswa 2000
+also suffixing most debugger commands with RUN will cause them not
+to stop just display the mnemonic at the current instruction on the console.
+If you have several breakpoints you want to put into your program &
+you get fed up of cross referencing with System.map
+you can do the following trick for several symbols.
+grep do_signal System.map 
+which emits the following among other things
+0001f4e0 T do_signal 
+now you can do
+
+TR I PSWA 0001f4e0 cmd msg * do_signal
+This sends a message to your own console each time do_signal is entered.
+( As an aside I wrote a perl script once which automatically generated a REXX
+script with breakpoints on every kernel procedure, this isn't a good idea
+because there are thousands of these routines & VM can only set 255 breakpoints
+at a time so you nearly had to spend as long pruning the file down as you would 
+entering the msg's by hand ),however, the trick might be useful for a single object file.
+On linux'es 3270 emulator x3270 there is a very useful option under the file ment
+Save Screens In File this is very good of keeping a copy of traces. 
+
+From CMS help <command name> will give you online help on a particular command. 
+e.g. 
+HELP DISPLAY
+
+Also CP has a file called profile.exec which automatically gets called
+on startup of CMS ( like autoexec.bat ), keeping on a DOS analogy session
+CP has a feature similar to doskey, it may be useful for you to
+use profile.exec to define some keystrokes. 
+e.g.
+SET PF9 IMM B
+This does a single step in VM on pressing F8. 
+SET PF10  ^
+This sets up the ^ key.
+which can be used for ^c (ctrl-c),^z (ctrl-z) which can't be typed directly into some 3270 consoles.
+SET PF11 ^-
+This types the starting keystrokes for a sysrq see SysRq below.
+SET PF12 RETRIEVE
+This retrieves command history on pressing F12.
+
+
+Sometimes in VM the display is set up to scroll automatically this
+can be very annoying if there are messages you wish to look at
+to stop this do
+TERM MORE 255 255
+This will nearly stop automatic screen updates, however it will
+cause a denial of service if lots of messages go to the 3270 console,
+so it would be foolish to use this as the default on a production machine.
+ 
+
+Tracing particular processes
+----------------------------
+The kernel's text segment is intentionally at an address in memory that it will
+very seldom collide with text segments of user programs ( thanks Martin ),
+this simplifies debugging the kernel.
+However it is quite common for user processes to have addresses which collide
+this can make debugging a particular process under VM painful under normal
+circumstances as the process may change when doing a 
+TR I R <address range>.
+Thankfully after reading VM's online help I figured out how to debug
+I particular process.
+
+Your first problem is to find the STD ( segment table designation )
+of the program you wish to debug.
+There are several ways you can do this here are a few
+1) objdump --syms <program to be debugged> | grep main
+To get the address of main in the program.
+tr i pswa <address of main>
+Start the program, if VM drops to CP on what looks like the entry
+point of the main function this is most likely the process you wish to debug.
+Now do a D X13 or D XG13 on z/Architecture.
+On 31 bit the STD is bits 1-19 ( the STO segment table origin ) 
+& 25-31 ( the STL segment table length ) of CR13.
+now type
+TR I R STD <CR13's value> 0.7fffffff
+e.g.
+TR I R STD 8F32E1FF 0.7fffffff
+Another very useful variation is
+TR STORE INTO STD <CR13's value> <address range>
+for finding out when a particular variable changes.
+
+An alternative way of finding the STD of a currently running process 
+is to do the following, ( this method is more complex but
+could be quite convient if you aren't updating the kernel much &
+so your kernel structures will stay constant for a reasonable period of
+time ).
+
+grep task /proc/<pid>/status
+from this you should see something like
+task: 0f160000 ksp: 0f161de8 pt_regs: 0f161f68
+This now gives you a pointer to the task structure.
+Now make CC:="s390-gcc -g" kernel/sched.s
+To get the task_struct stabinfo.
+( task_struct is defined in include/linux/sched.h ).
+Now we want to look at
+task->active_mm->pgd
+on my machine the active_mm in the task structure stab is
+active_mm:(4,12),672,32
+its offset is 672/8=84=0x54
+the pgd member in the mm_struct stab is
+pgd:(4,6)=*(29,5),96,32
+so its offset is 96/8=12=0xc
+
+so we'll
+hexdump -s 0xf160054 /dev/mem | more
+i.e. task_struct+active_mm offset
+to look at the active_mm member
+f160054 0fee cc60 0019 e334 0000 0000 0000 0011
+hexdump -s 0x0feecc6c /dev/mem | more
+i.e. active_mm+pgd offset
+feecc6c 0f2c 0000 0000 0001 0000 0001 0000 0010
+we get something like
+now do 
+TR I R STD <pgd|0x7f> 0.7fffffff
+i.e. the 0x7f is added because the pgd only
+gives the page table origin & we need to set the low bits
+to the maximum possible segment table length.
+TR I R STD 0f2c007f 0.7fffffff
+on z/Architecture you'll probably need to do
+TR I R STD <pgd|0x7> 0.ffffffffffffffff
+to set the TableType to 0x1 & the Table length to 3.
+
+
+
+Tracing Program Exceptions
+--------------------------
+If you get a crash which says something like
+illegal operation or specification exception followed by a register dump
+You can restart linux & trace these using the tr prog <range or value> trace option.
+
+
+
+The most common ones you will normally be tracing for is
+1=operation exception
+2=privileged operation exception
+4=protection exception
+5=addressing exception
+6=specification exception
+10=segment translation exception
+11=page translation exception
+
+The full list of these is on page 22 of the current s/390 Reference Summary.
+e.g.
+tr prog 10 will trace segment translation exceptions.
+tr prog on its own will trace all program interruption codes.
+
+Trace Sets
+----------
+On starting VM you are initially in the INITIAL trace set.
+You can do a Q TR to verify this.
+If you have a complex tracing situation where you wish to wait for instance 
+till a driver is open before you start tracing IO, but know in your
+heart that you are going to have to make several runs through the code till you
+have a clue whats going on. 
+
+What you can do is
+TR I PSWA <Driver open address>
+hit b to continue till breakpoint
+reach the breakpoint
+now do your
+TR GOTO B 
+TR IO 7c08-7c09 inst int run 
+or whatever the IO channels you wish to trace are & hit b
+
+To got back to the initial trace set do
+TR GOTO INITIAL
+& the TR I PSWA <Driver open address> will be the only active breakpoint again.
+
+
+Tracing linux syscalls under VM
+-------------------------------
+Syscalls are implemented on Linux for S390 by the Supervisor call instruction (SVC) there 256 
+possibilities of these as the instruction is made up of a  0xA opcode & the second byte being
+the syscall number. They are traced using the simple command.
+TR SVC  <Optional value or range>
+the syscalls are defined in linux/include/asm-s390/unistd.h
+e.g. to trace all file opens just do
+TR SVC 5 ( as this is the syscall number of open )
+
+
+SMP Specific commands
+---------------------
+To find out how many cpus you have
+Q CPUS displays all the CPU's available to your virtual machine
+To find the cpu that the current cpu VM debugger commands are being directed at do
+Q CPU to change the current cpu cpu VM debugger commands are being directed at do
+CPU <desired cpu no>
+
+On a SMP guest issue a command to all CPUs try prefixing the command with cpu all.
+To issue a command to a particular cpu try cpu <cpu number> e.g.
+CPU 01 TR I R 2000.3000
+If you are running on a guest with several cpus & you have a IO related problem
+& cannot follow the flow of code but you know it isnt smp related.
+from the bash prompt issue
+shutdown -h now or halt.
+do a Q CPUS to find out how many cpus you have
+detach each one of them from cp except cpu 0 
+by issuing a 
+DETACH CPU 01-(number of cpus in configuration)
+& boot linux again.
+TR SIGP will trace inter processor signal processor instructions.
+DEFINE CPU 01-(number in configuration) 
+will get your guests cpus back.
+
+
+Help for displaying ascii textstrings
+-------------------------------------
+On the very latest VM Nucleus'es VM can now display ascii
+( thanks Neale for the hint ) by doing
+D TX<lowaddr>.<len>
+e.g.
+D TX0.100
+
+Alternatively
+=============
+Under older VM debuggers ( I love EBDIC too ) you can use this little program I wrote which
+will convert a command line of hex digits to ascii text which can be compiled under linux & 
+you can copy the hex digits from your x3270 terminal to your xterm if you are debugging
+from a linuxbox.
+
+This is quite useful when looking at a parameter passed in as a text string
+under VM ( unless you are good at decoding ASCII in your head ).
+
+e.g. consider tracing an open syscall
+TR SVC 5
+We have stopped at a breakpoint
+000151B0' SVC   0A05     -> 0001909A'   CC 0
+
+D 20.8 to check the SVC old psw in the prefix area & see was it from userspace
+( for the layout of the prefix area consult P18 of the s/390 390 Reference Summary 
+if you have it available ).
+V00000020  070C2000 800151B2
+The problem state bit wasn't set &  it's also too early in the boot sequence
+for it to be a userspace SVC if it was we would have to temporarily switch the 
+psw to user space addressing so we could get at the first parameter of the open in
+gpr2.
+Next do a 
+D G2
+GPR  2 =  00014CB4
+Now display what gpr2 is pointing to
+D 00014CB4.20
+V00014CB4  2F646576 2F636F6E 736F6C65 00001BF5
+V00014CC4  FC00014C B4001001 E0001000 B8070707
+Now copy the text till the first 00 hex ( which is the end of the string
+to an xterm & do hex2ascii on it.
+hex2ascii 2F646576 2F636F6E 736F6C65 00 
+outputs
+Decoded Hex:=/ d e v / c o n s o l e 0x00 
+We were opening the console device,
+
+You can compile the code below yourself for practice :-),
+/*
+ *    hex2ascii.c
+ *    a useful little tool for converting a hexadecimal command line to ascii
+ *
+ *    Author(s): Denis Joseph Barrow (djbarrow@de.ibm.com,barrow_dj@yahoo.com)
+ *    (C) 2000 IBM Deutschland Entwicklung GmbH, IBM Corporation.
+ */   
+#include <stdio.h>
+
+int main(int argc,char *argv[])
+{
+  int cnt1,cnt2,len,toggle=0;
+  int startcnt=1;
+  unsigned char c,hex;
+  
+  if(argc>1&&(strcmp(argv[1],"-a")==0))
+     startcnt=2;
+  printf("Decoded Hex:=");
+  for(cnt1=startcnt;cnt1<argc;cnt1++)
+  {
+    len=strlen(argv[cnt1]);
+    for(cnt2=0;cnt2<len;cnt2++)
+    {
+       c=argv[cnt1][cnt2];
+       if(c>='0'&&c<='9')
+          c=c-'0';
+       if(c>='A'&&c<='F')
+          c=c-'A'+10;
+       if(c>='a'&&c<='f')
+          c=c-'a'+10;
+       switch(toggle)
+       {
+          case 0:
+             hex=c<<4;
+             toggle=1;
+          break;
+          case 1:
+             hex+=c;
+             if(hex<32||hex>127)
+             {
+                if(startcnt==1)
+                   printf("0x%02X ",(int)hex);
+                else
+                   printf(".");
+             }
+             else
+             {
+               printf("%c",hex);
+               if(startcnt==1)
+                  printf(" ");
+             }
+             toggle=0;
+          break;
+       }
+    }
+  }
+  printf("\n");
+}
+
+
+
+
+Stack tracing under VM
+----------------------
+A basic backtrace
+-----------------
+
+Here are the tricks I use 9 out of 10 times it works pretty well,
+
+When your backchain reaches a dead end
+--------------------------------------
+This can happen when an exception happens in the kernel & the kernel is entered twice
+if you reach the NULL pointer at the end of the back chain you should be
+able to sniff further back if you follow the following tricks.
+1) A kernel address should be easy to recognise since it is in
+primary space & the problem state bit isn't set & also
+The Hi bit of the address is set.
+2) Another backchain should also be easy to recognise since it is an 
+address pointing to another address approximately 100 bytes or 0x70 hex
+behind the current stackpointer.
+
+
+Here is some practice.
+boot the kernel & hit PA1 at some random time
+d g to display the gprs, this should display something like
+GPR  0 =  00000001  00156018  0014359C  00000000
+GPR  4 =  00000001  001B8888  000003E0  00000000
+GPR  8 =  00100080  00100084  00000000  000FE000
+GPR 12 =  00010400  8001B2DC  8001B36A  000FFED8
+Note that GPR14 is a return address but as we are real men we are going to
+trace the stack.
+display 0x40 bytes after the stack pointer.
+
+V000FFED8  000FFF38 8001B838 80014C8E 000FFF38
+V000FFEE8  00000000 00000000 000003E0 00000000
+V000FFEF8  00100080 00100084 00000000 000FE000
+V000FFF08  00010400 8001B2DC 8001B36A 000FFED8
+
+
+Ah now look at whats in sp+56 (sp+0x38) this is 8001B36A our saved r14 if
+you look above at our stackframe & also agrees with GPR14.
+
+now backchain 
+d 000FFF38.40
+we now are taking the contents of SP to get our first backchain.
+
+V000FFF38  000FFFA0 00000000 00014995 00147094
+V000FFF48  00147090 001470A0 000003E0 00000000
+V000FFF58  00100080 00100084 00000000 001BF1D0
+V000FFF68  00010400 800149BA 80014CA6 000FFF38
+
+This displays a 2nd return address of 80014CA6
+
+now do d 000FFFA0.40 for our 3rd backchain
+
+V000FFFA0  04B52002 0001107F 00000000 00000000
+V000FFFB0  00000000 00000000 FF000000 0001107F
+V000FFFC0  00000000 00000000 00000000 00000000
+V000FFFD0  00010400 80010802 8001085A 000FFFA0
+
+
+our 3rd return address is 8001085A
+
+as the 04B52002 looks suspiciously like rubbish it is fair to assume that the kernel entry routines
+for the sake of optimisation dont set up a backchain.
+
+now look at System.map to see if the addresses make any sense.
+
+grep -i 0001b3 System.map
+outputs among other things
+0001b304 T cpu_idle 
+so 8001B36A
+is cpu_idle+0x66 ( quiet the cpu is asleep, don't wake it )
+
+
+grep -i 00014 System.map 
+produces among other things
+00014a78 T start_kernel  
+so 0014CA6 is start_kernel+some hex number I can't add in my head.
+
+grep -i 00108 System.map 
+this produces
+00010800 T _stext
+so   8001085A is _stext+0x5a
+
+Congrats you've done your first backchain.
+
+
+
+s/390 & z/Architecture IO Overview
+==================================
+
+I am not going to give a course in 390 IO architecture as this would take me quite a
+while & I'm no expert. Instead I'll give a 390 IO architecture summary for Dummies if you have 
+the s/390 principles of operation available read this instead. If nothing else you may find a few 
+useful keywords in here & be able to use them on a web search engine like altavista to find 
+more useful information.
+
+Unlike other bus architectures modern 390 systems do their IO using mostly
+fibre optics & devices such as tapes & disks can be shared between several mainframes,
+also S390 can support upto 65536 devices while a high end PC based system might be choking 
+with around 64. Here is some of the common IO terminology
+
+Subchannel:
+This is the logical number most IO commands use to talk to an IO device there can be upto
+0x10000 (65536) of these in a configuration typically there is a few hundred. Under VM
+for simplicity they are allocated contiguously, however on the native hardware they are not
+they typically stay consistent between boots provided no new hardware is inserted or removed.
+Under Linux for 390 we use these as IRQ's & also when issuing an IO command (CLEAR SUBCHANNEL,
+HALT SUBCHANNEL,MODIFY SUBCHANNEL,RESUME SUBCHANNEL,START SUBCHANNEL,STORE SUBCHANNEL & 
+TEST SUBCHANNEL ) we use this as the ID of the device we wish to talk to, the most
+important of these instructions are START SUBCHANNEL ( to start IO ), TEST SUBCHANNEL ( to check
+whether the IO completed successfully ), & HALT SUBCHANNEL ( to kill IO ), a subchannel
+can have up to 8 channel paths to a device this offers redunancy if one is not available.
+
+
+Device Number:
+This number remains static & Is closely tied to the hardware, there are 65536 of these
+also they are made up of a CHPID ( Channel Path ID, the most significant 8 bits ) 
+& another lsb 8 bits. These remain static even if more devices are inserted or removed
+from the hardware, there is a 1 to 1 mapping between Subchannels & Device Numbers provided
+devices arent inserted or removed.
+
+Channel Control Words:
+CCWS are linked lists of instructions initially pointed to by an operation request block (ORB),
+which is initially given to Start Subchannel (SSCH) command along with the subchannel number
+for the IO subsystem to process while the CPU continues executing normal code.
+These come in two flavours, Format 0 ( 24 bit for backward )
+compatibility & Format 1 ( 31 bit ). These are typically used to issue read & write 
+( & many other instructions ) they consist of a length field & an absolute address field.
+For each IO typically get 1 or 2 interrupts one for channel end ( primary status ) when the
+channel is idle & the second for device end ( secondary status ) sometimes you get both
+concurrently, you check how the IO went on by issuing a TEST SUBCHANNEL at each interrupt,
+from which you receive an Interruption response block (IRB). If you get channel & device end 
+status in the IRB without channel checks etc. your IO probably went okay. If you didn't you
+probably need a doctorto examine the IRB & extended status word etc.
+If an error occurs more sophistocated control units have a facitity known as
+concurrent sense this means that if an error occurs Extended sense information will
+be presented in the Extended status word in the IRB if not you have to issue a
+subsequent SENSE CCW command after the test subchannel. 
+
+
+TPI( Test pending interrupt) can also be used for polled IO but in multitasking multiprocessor
+systems it isn't recommended except for checking special cases ( i.e. non looping checks for
+pending IO etc. ).
+
+Store Subchannel & Modify Subchannel can be used to examine & modify operating characteristics
+of a subchannel ( e.g. channel paths ).
+
+Other IO related Terms:
+Sysplex: S390's Clustering Technology
+QDIO: S390's new high speed IO architecture to support devices such as gigabit ethernet,
+this architecture is also designed to be forward compatible with up & coming 64 bit machines.
+
+
+General Concepts 
+
+Input Output Processors (IOP's) are responsible for communicating between
+the mainframe CPU's & the channel & relieve the mainframe CPU's from the
+burden of communicating with IO devices directly, this allows the CPU's to 
+concentrate on data processing. 
+
+IOP's can use one or more links ( known as channel paths ) to talk to each 
+IO device. It first checks for path availability & chooses an available one,
+then starts ( & sometimes terminates IO ).
+There are two types of channel path ESCON & the Paralell IO interface.
+
+IO devices are attached to control units, control units provide the
+logic to interface the channel paths & channel path IO protocols to 
+the IO devices, they can be integrated with the devices or housed separately
+& often talk to several similar devices ( typical examples would be raid 
+controllers or a control unit which connects to 1000 3270 terminals ).
+
+
+    +---------------------------------------------------------------+
+    | +-----+ +-----+ +-----+ +-----+  +----------+  +----------+   |
+    | | CPU | | CPU | | CPU | | CPU |  |  Main    |  | Expanded |   |
+    | |     | |     | |     | |     |  |  Memory  |  |  Storage |   |
+    | +-----+ +-----+ +-----+ +-----+  +----------+  +----------+   | 
+    |---------------------------------------------------------------+
+    |   IOP        |      IOP      |       IOP                      |
+    |---------------------------------------------------------------
+    | C | C | C | C | C | C | C | C | C | C | C | C | C | C | C | C | 
+    ----------------------------------------------------------------
+         ||                                              ||
+         ||  Bus & Tag Channel Path                      || ESCON
+         ||  ======================                      || Channel
+         ||  ||                  ||                      || Path
+    +----------+               +----------+         +----------+
+    |          |               |          |         |          |
+    |    CU    |               |    CU    |         |    CU    |
+    |          |               |          |         |          |
+    +----------+               +----------+         +----------+
+       |      |                     |                |       |
++----------+ +----------+      +----------+   +----------+ +----------+
+|I/O Device| |I/O Device|      |I/O Device|   |I/O Device| |I/O Device|
++----------+ +----------+      +----------+   +----------+ +----------+
+  CPU = Central Processing Unit    
+  C = Channel                      
+  IOP = IP Processor               
+  CU = Control Unit
+
+The 390 IO systems come in 2 flavours the current 390 machines support both
+
+The Older 360 & 370 Interface,sometimes called the paralell I/O interface,
+sometimes called Bus-and Tag & sometimes Original Equipment Manufacturers
+Interface (OEMI).
+
+This byte wide paralell channel path/bus has parity & data on the "Bus" cable 
+& control lines on the "Tag" cable. These can operate in byte multiplex mode for
+sharing between several slow devices or burst mode & monopolize the channel for the
+whole burst. Upto 256 devices can be addressed  on one of these cables. These cables are
+about one inch in diameter. The maximum unextended length supported by these cables is
+125 Meters but this can be extended up to 2km with a fibre optic channel extended 
+such as a 3044. The maximum burst speed supported is 4.5 megabytes per second however
+some really old processors support only transfer rates of 3.0, 2.0 & 1.0 MB/sec.
+One of these paths can be daisy chained to up to 8 control units.
+
+
+ESCON if fibre optic it is also called FICON 
+Was introduced by IBM in 1990. Has 2 fibre optic cables & uses either leds or lasers
+for communication at a signaling rate of upto 200 megabits/sec. As 10bits are transferred
+for every 8 bits info this drops to 160 megabits/sec & to 18.6 Megabytes/sec once
+control info & CRC are added. ESCON only operates in burst mode.
+ 
+ESCONs typical max cable length is 3km for the led version & 20km for the laser version
+known as XDF ( extended distance facility ). This can be further extended by using an
+ESCON director which triples the above mentioned ranges. Unlike Bus & Tag as ESCON is
+serial it uses a packet switching architecture the standard Bus & Tag control protocol
+is however present within the packets. Upto 256 devices can be attached to each control 
+unit that uses one of these interfaces.
+
+Common 390 Devices include:
+Network adapters typically OSA2,3172's,2116's & OSA-E gigabit ethernet adapters,
+Consoles 3270 & 3215 ( a teletype emulated under linux for a line mode console ).
+DASD's direct access storage devices ( otherwise known as hard disks ).
+Tape Drives.
+CTC ( Channel to Channel Adapters ),
+ESCON or Paralell Cables used as a very high speed serial link
+between 2 machines. We use 2 cables under linux to do a bi-directional serial link.
+
+
+Debugging IO on s/390 & z/Architecture under VM
+===============================================
+
+Now we are ready to go on with IO tracing commands under VM
+
+A few self explanatory queries:
+Q OSA
+Q CTC
+Q DISK ( This command is CMS specific )
+Q DASD
+
+
+
+
+
+
+Q OSA on my machine returns
+OSA  7C08 ON OSA   7C08 SUBCHANNEL = 0000
+OSA  7C09 ON OSA   7C09 SUBCHANNEL = 0001
+OSA  7C14 ON OSA   7C14 SUBCHANNEL = 0002
+OSA  7C15 ON OSA   7C15 SUBCHANNEL = 0003
+
+If you have a guest with certain priviliges you may be able to see devices
+which don't belong to you to avoid this do add the option V.
+e.g.
+Q V OSA
+
+Now using the device numbers returned by this command we will
+Trace the io starting up on the first device 7c08 & 7c09
+In our simplest case we can trace the 
+start subchannels
+like TR SSCH 7C08-7C09
+or the halt subchannels
+or TR HSCH 7C08-7C09
+MSCH's ,STSCH's I think you can guess the rest
+
+Ingo's favourite trick is tracing all the IO's & CCWS & spooling them into the reader of another
+VM guest so he can ftp the logfile back to his own machine.I'll do a small bit of this & give you
+ a look at the output.
+
+1) Spool stdout to VM reader
+SP PRT TO (another vm guest ) or * for the local vm guest
+2) Fill the reader with the trace
+TR IO 7c08-7c09 INST INT CCW PRT RUN
+3) Start up linux 
+i 00c  
+4) Finish the trace
+TR END
+5) close the reader
+C PRT
+6) list reader contents
+RDRLIST
+7) copy it to linux4's minidisk 
+RECEIVE / LOG TXT A1 ( replace
+8)
+filel & press F11 to look at it
+You should see someting like.
+
+00020942' SSCH  B2334000    0048813C    CC 0    SCH 0000    DEV 7C08
+          CPA 000FFDF0   PARM 00E2C9C4    KEY 0  FPI C0  LPM 80
+          CCW    000FFDF0  E4200100 00487FE8   0000  E4240100 ........
+          IDAL                                      43D8AFE8
+          IDAL                                      0FB76000
+00020B0A'   I/O DEV 7C08 -> 000197BC'   SCH 0000   PARM 00E2C9C4
+00021628' TSCH  B2354000 >> 00488164    CC 0    SCH 0000    DEV 7C08
+          CCWA 000FFDF8   DEV STS 0C  SCH STS 00  CNT 00EC
+           KEY 0   FPI C0  CC 0   CTLS 4007
+00022238' STSCH B2344000 >> 00488108    CC 0    SCH 0000    DEV 7C08
+
+If you don't like messing up your readed ( because you possibly booted from it )
+you can alternatively spool it to another readers guest.
+
+
+Other common VM device related commands
+---------------------------------------------
+These commands are listed only because they have
+been of use to me in the past & may be of use to
+you too. For more complete info on each of the commands
+use type HELP <command> from CMS.
+detaching devices
+DET <devno range>
+ATT <devno range> <guest> 
+attach a device to guest * for your own guest
+READY <devno> cause VM to issue a fake interrupt.
+
+The VARY command is normally only available to VM administrators.
+VARY ON PATH <path> TO <devno range>
+VARY OFF PATH <PATH> FROM <devno range>
+This is used to switch on or off channel paths to devices.
+
+Q CHPID <channel path ID>
+This displays state of devices using this channel path
+D SCHIB <subchannel>
+This displays the subchannel information SCHIB block for the device.
+this I believe is also only available to administrators.
+DEFINE CTC <devno>
+defines a virtual CTC channel to channel connection
+2 need to be defined on each guest for the CTC driver to use.
+COUPLE  devno userid remote devno
+Joins a local virtual device to a remote virtual device
+( commonly used for the CTC driver ).
+
+Building a VM ramdisk under CMS which linux can use
+def vfb-<blocksize> <subchannel> <number blocks>
+blocksize is commonly 4096 for linux.
+Formatting it
+format <subchannel> <driver letter e.g. x> (blksize <blocksize>
+
+Sharing a disk between multiple guests
+LINK userid devno1 devno2 mode password
+
+
+
+GDB on S390
+===========
+N.B. if compiling for debugging gdb works better without optimisation 
+( see Compiling programs for debugging )
+
+invocation
+----------
+gdb <victim program> <optional corefile>
+
+Online help
+-----------
+help: gives help on commands
+e.g.
+help
+help display
+Note gdb's online help is very good use it.
+
+
+Assembly
+--------
+info registers: displays registers other than floating point.
+info all-registers: displays floating points as well.
+disassemble: dissassembles
+e.g.
+disassemble without parameters will disassemble the current function
+disassemble $pc $pc+10 
+
+Viewing & modifying variables
+-----------------------------
+print or p: displays variable or register
+e.g. p/x $sp will display the stack pointer
+
+display: prints variable or register each time program stops
+e.g.
+display/x $pc will display the program counter
+display argc
+
+undisplay : undo's display's
+
+info breakpoints: shows all current breakpoints
+
+info stack: shows stack back trace ( if this dosent work too well, I'll show you the
+stacktrace by hand below ).
+
+info locals: displays local variables.
+
+info args: display current procedure arguments.
+
+set args: will set argc & argv each time the victim program is invoked.
+
+set <variable>=value
+set argc=100
+set $pc=0
+
+
+
+Modifying execution
+-------------------
+step: steps n lines of sourcecode
+step steps 1 line.
+step 100 steps 100 lines of code.
+
+next: like step except this will not step into subroutines
+
+stepi: steps a single machine code instruction.
+e.g. stepi 100
+
+nexti: steps a single machine code instruction but will not step into subroutines.
+
+finish: will run until exit of the current routine
+
+run: (re)starts a program
+
+cont: continues a program
+
+quit: exits gdb.
+
+
+breakpoints
+------------
+
+break
+sets a breakpoint
+e.g.
+
+break main
+
+break *$pc
+
+break *0x400618
+
+heres a really useful one for large programs
+rbr
+Set a breakpoint for all functions matching REGEXP
+e.g.
+rbr 390
+will set a breakpoint with all functions with 390 in their name.
+
+info breakpoints
+lists all breakpoints
+
+delete: delete breakpoint by number or delete them all
+e.g.
+delete 1 will delete the first breakpoint
+delete will delete them all
+
+watch: This will set a watchpoint ( usually hardware assisted ),
+This will watch a variable till it changes
+e.g.
+watch cnt, will watch the variable cnt till it changes.
+As an aside unfortunately gdb's, architecture independent watchpoint code
+is inconsistent & not very good, watchpoints usually work but not always.
+
+info watchpoints: Display currently active watchpoints
+
+condition: ( another useful one )
+Specify breakpoint number N to break only if COND is true.
+Usage is `condition N COND', where N is an integer and COND is an
+expression to be evaluated whenever breakpoint N is reached.
+
+
+
+User defined functions/macros
+-----------------------------
+define: ( Note this is very very useful,simple & powerful )
+usage define <name> <list of commands> end
+
+examples which you should consider putting into .gdbinit in your home directory
+define d
+stepi
+disassemble $pc $pc+10
+end
+
+define e
+nexti
+disassemble $pc $pc+10
+end
+
+
+Other hard to classify stuff
+----------------------------
+signal n:
+sends the victim program a signal.
+e.g. signal 3 will send a SIGQUIT.
+
+info signals:
+what gdb does when the victim receives certain signals.
+
+list:
+e.g.
+list lists current function source
+list 1,10 list first 10 lines of curret file.
+list test.c:1,10
+
+
+directory:
+Adds directories to be searched for source if gdb cannot find the source.
+(note it is a bit sensititive about slashes ) 
+e.g. To add the root of the filesystem to the searchpath do
+directory //
+
+
+call <function>
+This calls a function in the victim program, this is pretty powerful
+e.g.
+(gdb) call printf("hello world")
+outputs:
+$1 = 11 
+
+You might now be thinking that the line above didn't work, something extra had to be done.
+(gdb) call fflush(stdout)
+hello world$2 = 0
+As an aside the debugger also calls malloc & free under the hood 
+to make space for the "hello world" string.
+
+
+
+hints
+-----
+1) command completion works just like bash 
+( if you are a bad typist like me this really helps )
+e.g. hit br <TAB> & cursor up & down :-).
+
+2) if you have a debugging problem that takes a few steps to recreate
+put the steps into a file called .gdbinit in your current working directory
+if you have defined a few extra useful user defined commands put these in 
+your home directory & they will be read each time gdb is launched.
+
+A typical .gdbinit file might be.
+break main
+run
+break runtime_exception
+cont 
+
+
+stack chaining in gdb by hand
+-----------------------------
+This is done using a the same trick described for VM 
+p/x (*($sp+56))&0x7fffffff get the first backchain.
+
+For z/Architecture
+Replace 56 with 112 & ignore the &0x7fffffff
+in the macros below & do nasty casts to longs like the following
+as gdb unfortunately deals with printed arguments as ints which
+messes up everything.
+i.e. here is a 3rd backchain dereference
+p/x *(long *)(***(long ***)$sp+112)
+
+
+this outputs 
+$5 = 0x528f18 
+on my machine.
+Now you can use 
+info symbol (*($sp+56))&0x7fffffff 
+you might see something like.
+rl_getc + 36 in section .text  telling you what is located at address 0x528f18
+Now do.
+p/x (*(*$sp+56))&0x7fffffff 
+This outputs
+$6 = 0x528ed0
+Now do.
+info symbol (*(*$sp+56))&0x7fffffff
+rl_read_key + 180 in section .text
+now do
+p/x (*(**$sp+56))&0x7fffffff
+& so on.
+
+Disassembling instructions without debug info
+---------------------------------------------
+gdb typically compains if there is a lack of debugging
+symbols in  the disassemble command with 
+"No function contains specified address." to get around
+this do 
+x/<number lines to disassemble>xi <address>
+e.g.
+x/20xi 0x400730
+
+
+
+Note: Remember gdb has history just like bash you don't need to retype the
+whole line just use the up & down arrows.
+
+
+
+For more info
+-------------
+From your linuxbox do 
+man gdb or info gdb.
+
+core dumps
+----------
+What a core dump ?,
+A core dump is a file generated by the kernel ( if allowed ) which contains the registers,
+& all active pages of the program which has crashed.
+From this file gdb will allow you to look at the registers & stack trace & memory of the
+program as if it just crashed on your system, it is usually called core & created in the
+current working directory.
+This is very useful in that a customer can mail a core dump to a technical support department
+& the technical support department can reconstruct what happened.
+Provided the have an identical copy of this program with debugging symbols compiled in & 
+the source base of this build is available.
+In short it is far more useful than something like a crash log could ever hope to be.
+
+In theory all that is missing to restart a core dumped program is a kernel patch which
+will do the following.
+1) Make a new kernel task structure
+2) Reload all the dumped pages back into the kernel's memory management structures.
+3) Do the required clock fixups
+4) Get all files & network connections for the process back into an identical state ( really difficult ).
+5) A few more difficult things I haven't thought of.
+
+
+
+Why have I never seen one ?.
+Probably because you haven't used the command 
+ulimit -c unlimited in bash
+to allow core dumps, now do 
+ulimit -a 
+to verify that the limit was accepted.
+
+A sample core dump
+To create this I'm going to do
+ulimit -c unlimited
+gdb 
+to launch gdb (my victim app. ) now be bad & do the following from another 
+telnet/xterm session to the same machine
+ps -aux | grep gdb
+kill -SIGSEGV <gdb's pid>
+or alternatively use killall -SIGSEGV gdb if you have the killall command.
+Now look at the core dump.
+./gdb ./gdb core
+Displays the following
+GNU gdb 4.18
+Copyright 1998 Free Software Foundation, Inc.
+GDB is free software, covered by the GNU General Public License, and you are
+welcome to change it and/or distribute copies of it under certain conditions.
+Type "show copying" to see the conditions.
+There is absolutely no warranty for GDB.  Type "show warranty" for details.
+This GDB was configured as "s390-ibm-linux"...
+Core was generated by `./gdb'.
+Program terminated with signal 11, Segmentation fault.
+Reading symbols from /usr/lib/libncurses.so.4...done.
+Reading symbols from /lib/libm.so.6...done.
+Reading symbols from /lib/libc.so.6...done.
+Reading symbols from /lib/ld-linux.so.2...done.
+#0  0x40126d1a in read () from /lib/libc.so.6
+Setting up the environment for debugging gdb.
+Breakpoint 1 at 0x4dc6f8: file utils.c, line 471.
+Breakpoint 2 at 0x4d87a4: file top.c, line 2609.
+(top-gdb) info stack
+#0  0x40126d1a in read () from /lib/libc.so.6
+#1  0x528f26 in rl_getc (stream=0x7ffffde8) at input.c:402
+#2  0x528ed0 in rl_read_key () at input.c:381
+#3  0x5167e6 in readline_internal_char () at readline.c:454
+#4  0x5168ee in readline_internal_charloop () at readline.c:507
+#5  0x51692c in readline_internal () at readline.c:521
+#6  0x5164fe in readline (prompt=0x7ffff810 "\177�ÿ�øx\177�ÿ�÷�Ø\177�ÿ�øx�À")
+    at readline.c:349
+#7  0x4d7a8a in command_line_input (prrompt=0x564420 "(gdb) ", repeat=1,
+    annotation_suffix=0x4d6b44 "prompt") at top.c:2091
+#8  0x4d6cf0 in command_loop () at top.c:1345
+#9  0x4e25bc in main (argc=1, argv=0x7ffffdf4) at main.c:635
+
+
+LDD
+===
+This is a program which lists the shared libraries which a library needs,
+Note you also get the relocations of the shared library text segments which
+help when using objdump --source.
+e.g.
+ ldd ./gdb
+outputs
+libncurses.so.4 => /usr/lib/libncurses.so.4 (0x40018000)
+libm.so.6 => /lib/libm.so.6 (0x4005e000)
+libc.so.6 => /lib/libc.so.6 (0x40084000)
+/lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x40000000)
+
+
+Debugging shared libraries
+==========================
+Most programs use shared libraries, however it can be very painful
+when you single step instruction into a function like printf for the 
+first time & you end up in functions like _dl_runtime_resolve this is
+the ld.so doing lazy binding, lazy binding is a concept in ELF where 
+shared library functions are not loaded into memory unless they are 
+actually used, great for saving memory but a pain to debug.
+To get around this either relink the program -static or exit gdb type 
+export LD_BIND_NOW=true this will stop lazy binding & restart the gdb'ing 
+the program in question.
+ 
+
+
+Debugging modules
+=================
+As modules are dynamically loaded into the kernel their address can be
+anywhere to get around this use the -m option with insmod to emit a load
+map which can be piped into a file if required.
+
+The proc file system
+====================
+What is it ?.
+It is a filesystem created by the kernel with files which are created on demand
+by the kernel if read, or can be used to modify kernel parameters,
+it is a powerful concept.
+
+e.g.
+
+cat /proc/sys/net/ipv4/ip_forward 
+On my machine outputs 
+0 
+telling me ip_forwarding is not on to switch it on I can do
+echo 1 >  /proc/sys/net/ipv4/ip_forward
+cat it again
+cat /proc/sys/net/ipv4/ip_forward 
+On my machine now outputs
+1
+IP forwarding is on.
+There is a lot of useful info in here best found by going in & having a look around,
+so I'll take you through some entries I consider important.
+
+All the processes running on the machine have there own entry defined by
+/proc/<pid>
+So lets have a look at the init process
+cd /proc/1
+
+cat cmdline
+emits
+init [2]
+
+cd /proc/1/fd
+This contains numerical entries of all the open files,
+some of these you can cat e.g. stdout (2)
+
+cat /proc/29/maps
+on my machine emits
+
+00400000-00478000 r-xp 00000000 5f:00 4103       /bin/bash
+00478000-0047e000 rw-p 00077000 5f:00 4103       /bin/bash
+0047e000-00492000 rwxp 00000000 00:00 0
+40000000-40015000 r-xp 00000000 5f:00 14382      /lib/ld-2.1.2.so
+40015000-40016000 rw-p 00014000 5f:00 14382      /lib/ld-2.1.2.so
+40016000-40017000 rwxp 00000000 00:00 0
+40017000-40018000 rw-p 00000000 00:00 0
+40018000-4001b000 r-xp 00000000 5f:00 14435      /lib/libtermcap.so.2.0.8
+4001b000-4001c000 rw-p 00002000 5f:00 14435      /lib/libtermcap.so.2.0.8
+4001c000-4010d000 r-xp 00000000 5f:00 14387      /lib/libc-2.1.2.so
+4010d000-40111000 rw-p 000f0000 5f:00 14387      /lib/libc-2.1.2.so
+40111000-40114000 rw-p 00000000 00:00 0
+40114000-4011e000 r-xp 00000000 5f:00 14408      /lib/libnss_files-2.1.2.so
+4011e000-4011f000 rw-p 00009000 5f:00 14408      /lib/libnss_files-2.1.2.so
+7fffd000-80000000 rwxp ffffe000 00:00 0
+
+
+Showing us the shared libraries init uses where they are in memory
+& memory access permissions for each virtual memory area.
+
+/proc/1/cwd is a softlink to the current working directory.
+/proc/1/root is the root of the filesystem for this process. 
+
+/proc/1/mem is the current running processes memory which you
+can read & write to like a file.
+strace uses this sometimes as it is a bit faster than the
+rather inefficent ptrace interface for peeking at DATA.
+
+
+cat status 
+
+Name:   init
+State:  S (sleeping)
+Pid:    1
+PPid:   0
+Uid:    0       0       0       0
+Gid:    0       0       0       0
+Groups:
+VmSize:      408 kB
+VmLck:         0 kB
+VmRSS:       208 kB
+VmData:       24 kB
+VmStk:         8 kB
+VmExe:       368 kB
+VmLib:         0 kB
+SigPnd: 0000000000000000
+SigBlk: 0000000000000000
+SigIgn: 7fffffffd7f0d8fc
+SigCgt: 00000000280b2603
+CapInh: 00000000fffffeff
+CapPrm: 00000000ffffffff
+CapEff: 00000000fffffeff
+
+User PSW:    070de000 80414146
+task: 004b6000 tss: 004b62d8 ksp: 004b7ca8 pt_regs: 004b7f68
+User GPRS:
+00000400  00000000  0000000b  7ffffa90
+00000000  00000000  00000000  0045d9f4
+0045cafc  7ffffa90  7fffff18  0045cb08
+00010400  804039e8  80403af8  7ffff8b0
+User ACRS:
+00000000  00000000  00000000  00000000
+00000001  00000000  00000000  00000000
+00000000  00000000  00000000  00000000
+00000000  00000000  00000000  00000000
+Kernel BackChain  CallChain    BackChain  CallChain
+       004b7ca8   8002bd0c     004b7d18   8002b92c
+       004b7db8   8005cd50     004b7e38   8005d12a
+       004b7f08   80019114                     
+Showing among other things memory usage & status of some signals &
+the processes'es registers from the kernel task_structure
+as well as a backchain which may be useful if a process crashes
+in the kernel for some unknown reason.
+
+Some driver debugging techniques
+================================
+debug feature
+-------------
+Some of our drivers now support a "debug feature" in
+/proc/s390dbf see s390dbf.txt in the linux/Documentation directory
+for more info.
+e.g. 
+to switch on the lcs "debug feature"
+echo 5 > /proc/s390dbf/lcs/level
+& then after the error occurred.
+cat /proc/s390dbf/lcs/sprintf >/logfile
+the logfile now contains some information which may help
+tech support resolve a problem in the field.
+
+
+
+high level debugging network drivers
+------------------------------------
+ifconfig is a quite useful command
+it gives the current state of network drivers.
+
+If you suspect your network device driver is dead
+one way to check is type 
+ifconfig <network device> 
+e.g. tr0
+You should see something like
+tr0       Link encap:16/4 Mbps Token Ring (New)  HWaddr 00:04:AC:20:8E:48
+          inet addr:9.164.185.132  Bcast:9.164.191.255  Mask:255.255.224.0
+          UP BROADCAST RUNNING MULTICAST  MTU:2000  Metric:1
+          RX packets:246134 errors:0 dropped:0 overruns:0 frame:0
+          TX packets:5 errors:0 dropped:0 overruns:0 carrier:0
+          collisions:0 txqueuelen:100
+
+if the device doesn't say up
+try
+/etc/rc.d/init.d/network start 
+( this starts the network stack & hopefully calls ifconfig tr0 up ).
+ifconfig looks at the output of /proc/net/dev & presents it in a more presentable form
+Now ping the device from a machine in the same subnet.
+if the RX packets count & TX packets counts don't increment you probably
+have problems.
+next 
+cat /proc/net/arp
+Do you see any hardware addresses in the cache if not you may have problems.
+Next try
+ping -c 5 <broadcast_addr> i.e. the Bcast field above in the output of
+ifconfig. Do you see any replies from machines other than the local machine
+if not you may have problems. also if the TX packets count in ifconfig
+hasn't incremented either you have serious problems in your driver 
+(e.g. the txbusy field of the network device being stuck on ) 
+or you may have multiple network devices connected.
+
+
+chandev
+-------
+There is a new device layer for channel devices, some
+drivers e.g. lcs are registered with this layer.
+If the device uses the channel device layer you'll be
+able to find what interrupts it uses & the current state 
+of the device.
+See the manpage chandev.8 &type cat /proc/chandev for more info.
+
+
+
+Starting points for debugging scripting languages etc.
+======================================================
+
+bash/sh
+
+bash -x <scriptname>
+e.g. bash -x /usr/bin/bashbug
+displays the following lines as it executes them.
++ MACHINE=i586
++ OS=linux-gnu
++ CC=gcc
++ CFLAGS= -DPROGRAM='bash' -DHOSTTYPE='i586' -DOSTYPE='linux-gnu' -DMACHTYPE='i586-pc-linux-gnu' -DSHELL -DHAVE_CONFIG_H   -I. -I. -I./lib -O2 -pipe
++ RELEASE=2.01
++ PATCHLEVEL=1
++ RELSTATUS=release
++ MACHTYPE=i586-pc-linux-gnu   
+
+perl -d <scriptname> runs the perlscript in a fully intercative debugger
+<like gdb>.
+Type 'h' in the debugger for help.
+
+for debugging java type
+jdb <filename> another fully interactive gdb style debugger.
+& type ? in the debugger for help.
+
+
+
+Dumptool & Lcrash ( lkcd )
+==========================
+Michael Holzheu & others here at IBM have a fairly mature port of 
+SGI's lcrash tool which allows one to look at kernel structures in a
+running kernel.
+
+It also complements a tool called dumptool which dumps all the kernel's
+memory pages & registers to either a tape or a disk.
+This can be used by tech support or an ambitious end user do
+post mortem debugging of a machine like gdb core dumps.
+
+Going into how to use this tool in detail will be explained
+in other documentation supplied by IBM with the patches & the 
+lcrash homepage http://oss.sgi.com/projects/lkcd/ & the lcrash manpage.
+
+How they work
+-------------
+Lcrash is a perfectly normal program,however, it requires 2 
+additional files, Kerntypes which is built using a patch to the 
+linux kernel sources in the linux root directory & the System.map.
+
+Kerntypes is an an objectfile whose sole purpose in life
+is to provide stabs debug info to lcrash, to do this
+Kerntypes is built from kerntypes.c which just includes the most commonly
+referenced header files used when debugging, lcrash can then read the
+.stabs section of this file.
+
+Debugging a live system it uses /dev/mem
+alternatively for post mortem debugging it uses the data 
+collected by dumptool.
+
+
+
+SysRq
+=====
+This is now supported by linux for s/390 & z/Architecture.
+To enable it do compile the kernel with 
+Kernel Hacking -> Magic SysRq Key Enabled
+echo "1" > /proc/sys/kernel/sysrq
+also type
+echo "8" >/proc/sys/kernel/printk
+To make printk output go to console.
+On 390 all commands are prefixed with
+^-
+e.g.
+^-t will show tasks.
+^-? or some unknown command will display help.
+The sysrq key reading is very picky ( I have to type the keys in an
+ xterm session & paste them  into the x3270 console )
+& it may be wise to predefine the keys as described in the VM hints above
+
+This is particularly useful for syncing disks unmounting & rebooting
+if the machine gets partially hung.
+
+Read Documentation/sysrq.txt for more info
+
+References:
+===========
+Enterprise Systems Architecture Reference Summary
+Enterprise Systems Architecture Principles of Operation
+Hartmut Penners s390 stack frame sheet.
+IBM Mainframe Channel Attachment a technology brief from a CISCO webpage
+Various bits of man & info pages of Linux.
+Linux & GDB source.
+Various info & man pages.
+CMS Help on tracing commands.
+Linux for s/390 Elf Application Binary Interface
+Linux for z/Series Elf Application Binary Interface ( Both Highly Recommended )
+z/Architecture Principles of Operation SA22-7832-00
+Enterprise Systems Architecture/390 Reference Summary SA22-7209-01 & the
+Enterprise Systems Architecture/390 Principles of Operation SA22-7201-05
+
+Special Thanks
+==============
+Special thanks to Neale Ferguson who maintains a much
+prettier HTML version of this page at
+http://penguinvm.princeton.edu/notes.html#Debug390
+Bob Grainger Stefan Bader & others for reporting bugs