trace-cmd: Add documentation that will become manpages for trace-cmd

Added a Documentation directory that will become the trace-cmd manpages.

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>

10 files changed, 997 insertions, 0 deletions

diff --git a/Documentation/trace-cmd-extract.txt b/Documentation/trace-cmd-extract.txt
new file mode 100644
index 0000000..1eaea3c
--- /dev/null
+++ b/Documentation/trace-cmd-extract.txt
@@ -0,0 +1,66 @@
+TRACE-CMD-EXTRACT(1)
+===================
+
+NAME
+----
+trace-cmd-extract - extract out the data from the Ftrace Linux tracer.
+
+SYNOPSIS
+--------
+*trace-cmd extract ['OPTIONS']*
+
+DESCRIPTION
+-----------
+The trace-cmd(1) extract is usually used after 'trace-cmd-start(1)' and
+'trace-cmd-stop(1)'. It can be used after the Ftrace tracer has been started
+manually through the Ftrace pseudo file system.
+
+The extract command creates a trace.dat file that can be used by
+'trace-cmd-report(1)' to read from. It reads the kernel internal ring buffer
+to produce the trace.dat file.
+
+OPTIONS
+-------
+*-p* 'plugin'::
+    Although *extract* does not start any traces, some of the plugins require
+    just reading the output in ASCII format. These are the latency tracers,
+    since the latency tracers have a separate internal buffer. The plugin
+    option is therefore only necessary for the 'wakeup', 'wakeup-rt',
+    'irqsoff', 'preemptoff' and 'preemptirqsoff' plugins.
+
+    With out this option, the extract command will extract from the internal
+    Ftrace buffers.
+
+*-O* 'option'::
+    If a latency tracer is being extracted, and the *-p* option is used, then
+    there are some Ftrace options that can change the format. This will update
+    those options before extracting.  To see the list of options see
+    'trace-cmd-list'. To enable an option, write its name, to disable the
+    option append the characters 'no' to it. For example: 'noprint-parent'
+    will disable the 'print-parent' option that prints the parent function in
+    printing a function event.
+
+*-o* 'outputfile'::
+    By default, the extract command will create a 'trace.dat' file. This
+    option will change where the file is written to.
+
+
+SEE ALSO
+--------
+trace-cmd(1), trace-cmd-record(1), trace-cmd-report(1), trace-cmd-start(1),
+trace-cmd-stop(1), trace-cmd-reset(1), trace-cmd-split(1),
+trace-cmd-list(1), trace-cmd-listen(1)
+
+AUTHOR
+------
+Written by Steven Rostedt, <rostedt@goodmis.org>
+
+RESOURCES
+---------
+git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/trace-cmd.git
+
+COPYING
+-------
+Copyright \(C) 2010 Red Hat, Inc. Free use of this software is granted under
+the terms of the GNU Public License (GPL).
+
diff --git a/Documentation/trace-cmd-list.txt b/Documentation/trace-cmd-list.txt
new file mode 100644
index 0000000..16d0b6b
--- /dev/null
+++ b/Documentation/trace-cmd-list.txt
@@ -0,0 +1,51 @@
+TRACE-CMD-LIST(1)
+==================
+
+NAME
+----
+trace-cmd-list - list available plugins, events or options for Ftrace.
+
+SYNOPSIS
+--------
+*trace-cmd list* ['OPTIONS']
+
+DESCRIPTION
+-----------
+The trace-cmd(1) list displays the available plugins, events or Ftrace options
+that are configured on the current machine.  If no option is given, then it
+lists all plugins, events and Ftrace options to standard output.
+
+OPTIONS
+-------
+*-e*::
+    This option will list the available events that are enabled on the
+    local system.
+
+*-p*::
+    This option will list the available plugins that are enabled on the
+    local system.
+
+*-o*::
+    This option will list the available Ftrace options that are configured on
+    the local system.
+
+
+SEE ALSO
+--------
+trace-cmd(1), trace-cmd-record(1), trace-cmd-report(1), trace-cmd-start(1),
+trace-cmd-stop(1), trace-cmd-extract(1), trace-cmd-reset(1),
+trace-cmd-split(1), trace-cmd-listen(1)
+
+AUTHOR
+------
+Written by Steven Rostedt, <rostedt@goodmis.org>
+
+RESOURCES
+---------
+git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/trace-cmd.git
+
+COPYING
+-------
+Copyright \(C) 2010 Red Hat, Inc. Free use of this software is granted under
+the terms of the GNU Public License (GPL).
+
diff --git a/Documentation/trace-cmd-listen.txt b/Documentation/trace-cmd-listen.txt
new file mode 100644
index 0000000..981fb96
--- /dev/null
+++ b/Documentation/trace-cmd-listen.txt
@@ -0,0 +1,55 @@
+TRACE-CMD-LISTEN(1)
+==================
+
+NAME
+----
+trace-cmd-listen - listen for incoming connection to record tracing.
+
+SYNOPSIS
+--------
+*trace-cmd listen* -p 'port' ['OPTIONS']
+
+DESCRIPTION
+-----------
+The trace-cmd(1) listen sets up a port to listen to waiting for connections
+from other hosts that run 'trace-cmd-record(1)' with the *-N* option. When a
+connection is made, and the remote host sends data, it will create a file
+called 'trace.HOST:PORT.dat'. Where HOST is the name of the remote host, and
+PORT is the port that the remote host used to connect with.
+
+OPTIONS
+-------
+*-p* 'port'::
+    This option will specifies tha port to listen to.
+
+*-D*::
+    This options causes trace-cmd listen to go into a daemon mode and run in
+    the background.
+
+*-d* 'dir'::
+    This option specifies a directory to write the data files into.
+
+*-o* 'filename'::
+    This option overides the default 'trace' in the 'trace.HOST:PORT.dat' that
+    is created when a remote host connects.
+
+
+SEE ALSO
+--------
+trace-cmd(1), trace-cmd-record(1), trace-cmd-report(1), trace-cmd-start(1),
+trace-cmd-stop(1), trace-cmd-extract(1), trace-cmd-reset(1),
+trace-cmd-split(1), trace-cmd-list(1)
+
+AUTHOR
+------
+Written by Steven Rostedt, <rostedt@goodmis.org>
+
+RESOURCES
+---------
+git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/trace-cmd.git
+
+COPYING
+-------
+Copyright \(C) 2010 Red Hat, Inc. Free use of this software is granted under
+the terms of the GNU Public License (GPL).
+
diff --git a/Documentation/trace-cmd-record.txt b/Documentation/trace-cmd-record.txt
new file mode 100644
index 0000000..9f43d6d
--- /dev/null
+++ b/Documentation/trace-cmd-record.txt
@@ -0,0 +1,225 @@
+TRACE-CMD-RECORD(1)
+===================
+
+NAME
+----
+trace-cmd-record - record a trace from the Ftrace Linux internal tracer
+
+SYNOPSIS
+--------
+*trace-cmd record* ['OPTIONS'] ['command']
+
+DESCRIPTION
+-----------
+The trace-cmd(1) record command will set up the Ftrace Linux kernel tracer to
+record the specified plugins or events that happen while the 'command'
+executes. If no command is given, then it will record until the user hits
+Ctrl-C.
+
+The record command of trace-cmd will set up the Ftrace tracer to start tracing
+the various events or plugins that are given on the command line. It will then
+create a number of tracing processes (one per CPU) that will start recording
+from the kernel ring buffer straight into temporary files. When the command is
+complete (or Ctrl-C is hit) all the files will be combined into a trace.dat
+file that can later be read (see trace-cmd-report(1)).
+
+OPTIONS
+-------
+*-p* 'plugin'::
+    Specify a trace plugin. Plugins are special Ftrace tracers that usually do
+    more than just trace an event. Common plugins are *function*,
+    *function_graph*, *preemptirqsoff*, *irqsoff*, *preemptoff*, and *wakeup*.
+    A plugin must be supported by the running kernel. To see a list of
+    available plugins, see trace-cmd-list(1).
+
+*-e* 'event'::
+    Specify an event to trace. Various static trace points have been added to
+    the Linux kernel. They are grouped by subsystem where you can enable all
+    events of a given subsystem or specify specific events to be enabled. The
+    'event' is of the format "subsystem:event-name". You can also just specify
+    the subsystem without the ':event-name' or the eventname without the
+    "subsystem:". Using "-e sched_switch" will enable the "sched_switch" event
+    where as, "-e sched" will enable all events under the "sched" subsystem.
+
+    The 'event' can also contain glob expressions. That is, "*stat*" will
+    select all events (or subsystems) that have the characters "stat" in their
+    names.
+
+    The keyword 'all' can be used to enable all events.
+
+*-f* 'filter'::
+    Specify a filter for the previous event. This must come after a *-e*. This
+    will filter what events get recorded based on the content of the event.
+    Filtering is passed to the kernel directly so what filtering is allowed
+    may depend on what verison of the kernel you have. Basically, it will
+    let you use C notation to check if an event should be processed or not.
+
+----------------------------------------
+    ==, >=, <=, >, <, &, |, && and ||
+----------------------------------------
+
+    The above are usually safe to use to compare fields.
+
+*-v*::
+    This will cause all events specified after it on the command line to not
+    be traced. This is useful for selecting a subsystem to be traced but to
+    leave out various events. For Example: "-e sched -v -e "\*stat\*"" will
+    enable all events in the sched subsystem except those that have "stat" in
+    their names.
+
+    Note: the *-v* option was taken from the way grep(1) inverts the following
+    matches.
+
+*-F*::
+    This will filter only the executable that is given on the command line. If
+    no command is given, then it will filter itself (pretty pointless).
+    Using *-F* will let you trace only events that are caused by the given
+    command.
+
+*-P* 'pid'::
+     Similar to *-F* but lets you specify a process ID to trace.
+
+*-o* 'output-file'::
+    By default, trace-cmd report will create a 'trace.dat' file. You can
+    specify a different file to write to with the *-o* option.
+
+*-l* 'function-name'::
+    This will limit the 'function' and 'function_graph' tracers to only trace
+    the given function name. More than one *-l* may be specified on the
+    command line to trace more than one function. The limited use of glob
+    expressions are also allowed. These are 'match\*' to only filter functions
+    that start with 'match'. '\*match' to only filter functions that end with
+    'match'. '\*match\*' to only filter on functions that contain 'match'.
+
+*-g* 'function-name'::
+    This option is for the function_graph plugin. It will graph the given
+    function. That is, it will only trace the function and all functions that
+    it calls. You can have more than one *-g* on the command line.
+
+*-n* 'function-name'::
+    This has the opposite effect of *-l*. The function given with the *-n*
+    option will not be traced. This takes precedence, that is, if you include
+    the same function for both *-n* and *-l*, it will not be traced.
+
+*-d*::
+    Some tracer plugins enable the function tracer by default. Like the
+    latency tracers. This option prevents the function tracer from being
+    enabled at start up.
+
+*-O* 'option'::
+    Ftrace has various options that can be enabled or disabled. This allows
+    you to set them. Appending the text 'no' to an option disables it.
+    For example: "-O nograph-time" will disable the "graph-time" Ftrace
+    option.
+
+*-s* 'interval'::
+    The processes that trace-cmd creates to record from the ring buffer need
+    to wake up to do the recording. Setting the 'interval' to zero will cause
+    the processes to wakeup every time new data is written into the buffer.
+    But since Ftrace is recording kernel activity, the act of this processes
+    going back to sleep may cause new events into the ring buffer which will
+    wake the process back up. This will needlessly add extra data into the
+    ring buffer.
+
+    The 'interval' metric is microseconds. The default is set to 1000 (1 ms).
+    This is the time each recording process will sleep before waking up to
+    record any new data that was written to the ring buffer.
+
+*-b* 'size'::
+    This sets the ring buffer size to 'size' kilobytes. Because the Ftrace
+    ring buffer is per CPU, this size is the size of each per CPU ring buffer
+    inside the kernel. Using "-b 10000" on a machine with 4 CPUs will make
+    Ftrace have a total buffer size of 40 Megs.
+
+*-N* 'host:port'::
+    If another machine is running "trace-cmd listen", this option is used to
+    have the data sent to that machine with UDP packets. Instead of writing
+    to an output file, the data is sent off to a remote box. This is ideal for
+    embedded machines with little storage, or having a single machine that
+    will keep all the data in a single repository.
+
+*-t*::
+    This option is used with *-N*, when there's a need to send the live data
+    with TCP packets instead of UDP. Although TCP is not nearly as fast as
+    sending the UDP packets, but it may be needed if the network is not that
+    reliable, the amount of data is not that intensive, and a guarantee is
+    needed that all traced information is transfered successfully.
+
+
+EXAMPLES
+--------
+
+The basic way to trace all events:
+
+------------------------------
+ # trace-cmd record -e all ls > /dev/null
+ # trace-cmd report
+       trace-cmd-13541 [003] 106260.693809: filemap_fault: address=0x128122 offset=0xce
+       trace-cmd-13543 [001] 106260.693809: kmalloc: call_site=81128dd4 ptr=0xffff88003dd83800 bytes_req=768 bytes_alloc=1024 gfp_flags=GFP_KERNEL|GFP_ZERO
+              ls-13545 [002] 106260.693809: kfree: call_site=810a7abb ptr=0x0
+              ls-13545 [002] 106260.693818: sys_exit_write:       0x1
+
+
+------------------------------
+
+
+
+To use the function tracer with sched switch tracing:
+
+------------------------------
+ # trace-cmd record -p function -e sched_switch ls > /dev/null
+ # trace-cmd report
+              ls-13587 [002] 106467.860310: function: hrtick_start_fair <-- pick_next_task_fair
+              ls-13587 [002] 106467.860313: sched_switch: prev_comm=trace-cmd prev_pid=13587 prev_prio=120 prev_state=R ==> next_comm=trace-cmd next_pid=13583 next_prio=120
+       trace-cmd-13585 [001] 106467.860314: function: native_set_pte_at <-- __do_fault
+       trace-cmd-13586 [003] 106467.860314: function:             up_read <-- do_page_fault
+              ls-13587 [002] 106467.860317: function:             __phys_addr <-- schedule
+       trace-cmd-13585 [001] 106467.860318: function: _raw_spin_unlock <-- __do_fault
+              ls-13587 [002] 106467.860320: function: native_load_sp0 <-- __switch_to
+       trace-cmd-13586 [003] 106467.860322: function: down_read_trylock <-- do_page_fault
+
+
+------------------------------
+
+Here is a nice way to find what interrupts have the highest latency:
+------------------------------------------
+ # trace-cmd record -p function_graph -e irq_handler_entry  -l do_IRQ sleep 10
+ # trace-cmd report
+          <idle>-0     [000] 157412.933969: funcgraph_entry:                  |  do_IRQ() {
+          <idle>-0     [000] 157412.933974: irq_handler_entry:    irq=48 name=eth0
+          <idle>-0     [000] 157412.934004: funcgraph_exit:       + 36.358 us |  }
+          <idle>-0     [000] 157413.895004: funcgraph_entry:                  |  do_IRQ() {
+          <idle>-0     [000] 157413.895011: irq_handler_entry:    irq=48 name=eth0
+          <idle>-0     [000] 157413.895026: funcgraph_exit:                        + 24.014 us |  }
+          <idle>-0     [000] 157415.891762: funcgraph_entry:                  |  do_IRQ() {
+          <idle>-0     [000] 157415.891769: irq_handler_entry:    irq=48 name=eth0
+          <idle>-0     [000] 157415.891784: funcgraph_exit:       + 22.928 us |  }
+          <idle>-0     [000] 157415.934869: funcgraph_entry:                  |  do_IRQ() {
+          <idle>-0     [000] 157415.934874: irq_handler_entry:    irq=48 name=eth0
+          <idle>-0     [000] 157415.934906: funcgraph_exit:       + 37.512 us |  }
+          <idle>-0     [000] 157417.888373: funcgraph_entry:                  |  do_IRQ() {
+          <idle>-0     [000] 157417.888381: irq_handler_entry:    irq=48 name=eth0
+          <idle>-0     [000] 157417.888398: funcgraph_exit:       + 25.943 us |  }
+
+
+------------------------------------------
+
+SEE ALSO
+--------
+trace-cmd(1), trace-cmd-report(1), trace-cmd-start(1), trace-cmd-stop(1),
+trace-cmd-extract(1), trace-cmd-reset(1), trace-cmd-split(1),
+trace-cmd-list(1), trace-cmd-listen(1)
+
+AUTHOR
+------
+Written by Steven Rostedt, <rostedt@goodmis.org>
+
+RESOURCES
+---------
+git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/trace-cmd.git
+
+COPYING
+-------
+Copyright \(C) 2010 Red Hat, Inc. Free use of this software is granted under
+the terms of the GNU Public License (GPL).
+
diff --git a/Documentation/trace-cmd-report.txt b/Documentation/trace-cmd-report.txt
new file mode 100644
index 0000000..aa033f5
--- /dev/null
+++ b/Documentation/trace-cmd-report.txt
@@ -0,0 +1,285 @@
+TRACE-CMD-REPORT(1)
+===================
+
+NAME
+----
+trace-cmd-report - show in ASCII a trace created by trace-cmd record
+
+SYNOPSIS
+--------
+*trace-cmd report* ['OPTIONS'] ['input-file']
+
+DESCRIPTION
+-----------
+The trace-cmd(1) report command will output a human readable report of a trace
+created by trace-cmd record.
+
+OPTIONS
+-------
+*-i* 'input-file'::
+    By default, trace-cmd report willl read the file 'trace.dat'. But the *-i*
+    option open up the given 'input-file' instead. Note, the input file may
+    also be specified as the last item on the command line.
+
+*-e*::
+    This outputs the endianess of the file. trace-cmd report is smart enough
+    to be able to read big endian files on little endian machines, and vise
+    versa.
+
+*-f*::
+    This outputs the list of functions that have been recorded in the file.
+
+*-P*::
+    This outputs the list of "trace_printk()" data. The raw trace data points
+    to static pointers in the kernel. This must be stored in the trace.dat
+    file.
+
+*-E*::
+    This lists the possible events in the file (but this list is not
+    necessarily the list of events in the file).
+
+*--events*::
+    This will list the event formats that are stored in the trace.dat file.
+
+*-F* 'filter'::
+    Add a filter to limit what events are displayed. The format of the filter
+    is:
+
+------------------------------------------
+    <events> ':' <filter>
+    <events> = SYSTEM'/'EVENT  | SYSTEM | EVENT | <events> ',' <events>
+    <filter> = EVENT_FIELD <op> <value> | <filter> '&&' <filter> |
+               <filter> '||' <filter> | '(' <filter> ')' | '!' <filter>
+    <op> = '==' | '!=' | '>=' | '<=' | '>' | '<' | '&' | '|' | '^' |
+           '+' | '-' | '*' | '/' | '%'
+    <value> = NUM | STRING | EVENT_FIELD
+------------------------------------------
+
+    SYSTEM is the name of the system to filter on. If the EVENT is left out,
+    then it applies to all events under the SYSTEM. If only one string is used
+    without the '/' to deliminate between SYSTEM and EVENT, then the filter
+    will be applied to all systems and events that match the given string.
+
+    Whitespace is ignored, such that "sched:next_pid==123" is equivalent to
+    "sched : next_pid == 123".
+
+    STRING is defined with single or double quotes (single quote must end with
+    single quote, and double with double). Whitespace within quotes are not
+    ignored.
+
+    The representation of a SYSTEM or EVENT may also be a regular expression
+    as defined by 'regcomp(3)'.
+
+    The EVENT_FIELD is the name of the field of an event that is being
+    filtered. If the event does not contain the EVENT_FIELD, that part of the
+    equation will be considered false.
+
+------------------------------------------
+    -F 'sched : bogus == 1 || common_pid == 2'
+------------------------------------------
+
+    The "bogus == 1" will always evaluate to FALSE because no event has a
+    field called "bogus", but the "common_pid == 2" will still be evaluated
+    since all events have the field "common_pid". Any "sched" event that was
+    traced by the process with the PID of 2 will be shown.
+
+    Note, the EVENT_FIELD is the field name as shown by an events format
+    (as displayed with *--events*), and not what is found in the output.
+    If the output shows "ID:foo" but the field that "foo" belongs to was
+    called "name" in the event format, then "name" must be used in the filter.
+    The same is true about values. If the value that is displayed is converted
+    by to a string symbol, the filter checks the original value and not the
+    value displayed. For example, to filter on all tasks that were in the
+    running state at a context switch:
+
+------------------------------------------
+    -F 'sched/sched_switch : prev_state==0'
+------------------------------------------
+
+    Although the output displays 'R', having 'prev_stat=="R"' will not work.
+
+*-v*::
+    This causes the following filters of *-F* to filter out the matching
+    events.
+
+------------------------------------------
+    -v -F 'sched/sched_switch : prev_state == 0'
+------------------------------------------
+
+    Will not display any sched_switch events that have a prev_state of 0.
+    Removing the *-v* will only print out those events.
+
+*-l*::
+    This adds a "latency output" format. Information about interrupts being
+    disabled, soft irq being disabled, the "need_resched" flag being set,
+    preempt count, and big kernel lock are all being recorded with every
+    event. But the default display does not show this information. This option
+    will set display this information with 6 characters. When one of the
+    fields is zero or N/A a \'.\' is shown.
+
+------------------------------------------
+      <idle>-0       0d.h1. 106467.859747: function:             ktime_get <-- tick_check_idle
+------------------------------------------
+
+    The 0d.h1. denotes this information. The first character is never a '.'
+    and represents what CPU the trace was recorded on (CPU 0). The 'd' denotes
+    that interrupts were disabled. The 'h' means that this was called inside
+    an interrupt handler. The '1' is the preemption disabled (preempt_count)
+    was set to one.  The two '.'s are "need_resched" flag and kernel lock
+    counter.  If the "need_resched" flag is set, then that charater would be a
+    'N'.
+
+*-w*::
+    If both the 'sched_switch' and 'sched_wakeup' events are enabled, then
+    this option will report the latency between the time the task was first
+    woken, and the time it was scheduled in.
+
+EXAMPLES
+--------
+
+Using a trace.dat file that was created with:
+
+------------------------------------------
+    # trace-cmd record -p function -e all sleep 5
+
+
+------------------------------------------
+
+The default report shows:
+
+------------------------------------------
+ # trace-cmd report
+       trace-cmd-16129 [002] 158126.498411: function: __mutex_unlock_slowpath <-- mutex_unlock
+       trace-cmd-16131 [000] 158126.498411: kmem_cache_alloc: call_site=811223c5 ptr=0xffff88003ecf2b40 bytes_req=272 bytes_alloc=320 gfp_flags=GFP_KERNEL|GFP_ZERO
+       trace-cmd-16130 [003] 158126.498411: function:             do_splice_to <-- sys_splice
+           sleep-16133 [001] 158126.498412: function: inotify_inode_queue_event <-- vfs_write
+       trace-cmd-16129 [002] 158126.498420: lock_release: 0xffff88003f1fa4f8 &sb->s_type->i_mutex_key
+       trace-cmd-16131 [000] 158126.498421: function: security_file_alloc <-- get_empty_filp
+           sleep-16133 [001] 158126.498422: function: __fsnotify_parent <-- vfs_write
+       trace-cmd-16130 [003] 158126.498422: function: rw_verify_area <-- do_splice_to
+       trace-cmd-16131 [000] 158126.498424: function: cap_file_alloc_security <-- security_file_alloc
+       trace-cmd-16129 [002] 158126.498425: function: syscall_trace_leave <-- int_check_syscall_exit_work
+           sleep-16133 [001] 158126.498426: function: inotify_dentry_parent_queue_event <-- vfs_write
+       trace-cmd-16130 [003] 158126.498426: function: security_file_permission <-- rw_verify_area
+       trace-cmd-16129 [002] 158126.498428: function: audit_syscall_exit <-- syscall_trace_leave
+[...]
+
+
+------------------------------------------
+
+To see everything but the function traces:
+
+------------------------------------------
+ # trace-cmd report -v -F 'function'
+       trace-cmd-16131 [000] 158126.498411: kmem_cache_alloc: call_site=811223c5 ptr=0xffff88003ecf2b40 bytes_req=272 bytes_alloc=320 gfp_flags=GFP_KERNEL|GFP_ZERO
+       trace-cmd-16129 [002] 158126.498420: lock_release: 0xffff88003f1fa4f8 &sb->s_type->i_mutex_key
+       trace-cmd-16130 [003] 158126.498436: lock_acquire: 0xffffffff8166bf78 read all_cpu_access_lock
+       trace-cmd-16131 [000] 158126.498438: lock_acquire: 0xffff88003df5b520 read &fs->lock
+       trace-cmd-16129 [002] 158126.498446: kfree: call_site=810a7abb ptr=0x0
+       trace-cmd-16130 [003] 158126.498448: lock_acquire: 0xffff880002250a80 &per_cpu(cpu_access_lock, cpu)
+       trace-cmd-16129 [002] 158126.498450: sys_exit_splice:      0xfffffff5
+       trace-cmd-16131 [000] 158126.498454: lock_release: 0xffff88003df5b520 &fs->lock
+           sleep-16133 [001] 158126.498456: kfree: call_site=810a7abb ptr=0x0
+           sleep-16133 [001] 158126.498460: sys_exit_write:       0x1
+       trace-cmd-16130 [003] 158126.498462: kmalloc: call_site=810bf95b ptr=0xffff88003dedc040 bytes_req=24 bytes_alloc=32 gfp_flags=GFP_KERNEL|GFP_ZERO
+
+
+------------------------------------------
+
+To see only the kmalloc calls that were greater than 1000 bytes:
+
+------------------------------------------
+ #trace-cmd report -F 'kmalloc: bytes_req > 1000'
+          <idle>-0     [000] 158128.126641: kmalloc: call_site=81330635 ptr=0xffff88003c2fd000 bytes_req=2096 bytes_alloc=4096 gfp_flags=GFP_ATOMIC
+
+
+------------------------------------------
+
+To see wakeups and sched switches that left the previous task in the running
+state:
+------------------------------------------
+ # trace-cmd report -F 'sched: prev_state == 0 || (success == 1)'
+       trace-cmd-16132 [002] 158126.499951: sched_wakeup: comm=trace-cmd pid=16129 prio=120 success=1 target_cpu=002
+       trace-cmd-16132 [002] 158126.500401: sched_switch: prev_comm=trace-cmd prev_pid=16132 prev_prio=120 prev_state=R ==> next_comm=trace-cmd next_pid=16129 next_prio=120
+          <idle>-0     [003] 158126.500585: sched_wakeup: comm=trace-cmd pid=16130 prio=120 success=1 target_cpu=003
+          <idle>-0     [003] 158126.501241: sched_switch: prev_comm=swapper prev_pid=0 prev_prio=120 prev_state=R ==> next_comm=trace-cmd next_pid=16130 next_prio=120
+       trace-cmd-16132 [000] 158126.502475: sched_wakeup: comm=trace-cmd pid=16131 prio=120 success=1 target_cpu=000
+       trace-cmd-16131 [002] 158126.506516: sched_wakeup: comm=trace-cmd pid=16129 prio=120 success=1 target_cpu=002
+          <idle>-0     [003] 158126.550110: sched_switch: prev_comm=swapper prev_pid=0 prev_prio=120 prev_state=R ==> next_comm=trace-cmd next_pid=16130 next_prio=120
+       trace-cmd-16131 [003] 158126.570243: sched_wakeup: comm=trace-cmd pid=16129 prio=120 success=1 target_cpu=003
+       trace-cmd-16130 [002] 158126.618202: sched_switch: prev_comm=trace-cmd prev_pid=16130 prev_prio=120 prev_state=R ==> next_comm=yum-updatesd next_pid=3088 next_prio=1 20
+       trace-cmd-16129 [003] 158126.622379: sched_wakeup: comm=trace-cmd pid=16131 prio=120 success=1 target_cpu=003
+       trace-cmd-16129 [000] 158126.649287: sched_wakeup: comm=trace-cmd pid=16131 prio=120 success=1 target_cpu=000
+
+
+------------------------------------------
+
+The above needs a little explanation. The filter specifies the "sched"
+subsystem, which includes both sched_switch and sched_wakeup events. Any event
+that does not have the format field "prev_state" or "success", will evaluate
+those expressions as FALSE, and will not produce a match. Using "||" will have
+the "prev_state" test happen for the "sched_switch" event and the "success"
+test happen for the "sched_wakeup" event.
+
+
+------------------------------------------
+  # trace-cmd report -w -F 'sched_switch, sched_wakeup.*'
+[...]
+       trace-cmd-16130 [003] 158131.580616: sched_wakeup: comm=trace-cmd pid=16131 prio=120 success=1 target_cpu=003
+       trace-cmd-16129 [000] 158131.581502: sched_switch: prev_comm=trace-cmd prev_pid=16129 prev_prio=120 prev_state=S ==> next_comm=trace-cmd next_pid=16131 next_prio=120 Latency: 885.901 usecs
+       trace-cmd-16131 [000] 158131.582414: sched_wakeup: comm=trace-cmd pid=16129 prio=120 success=1 target_cpu=000
+       trace-cmd-16132 [001] 158131.583219: sched_switch: prev_comm=trace-cmd prev_pid=16132 prev_prio=120 prev_state=S ==> next_comm=trace-cmd next_pid=16129 next_prio=120 Latency: 804.809 usecs
+           sleep-16133 [002] 158131.584121: sched_wakeup: comm=trace-cmd pid=16120 prio=120 success=1 target_cpu=002
+       trace-cmd-16129 [001] 158131.584128: sched_wakeup: comm=trace-cmd pid=16132 prio=120 success=1 target_cpu=001
+           sleep-16133 [002] 158131.584275: sched_switch: prev_comm=sleep prev_pid=16133 prev_prio=120 prev_state=R ==> next_comm=trace-cmd next_pid=16120 next_prio=120 Latency: 153.915 usecs
+       trace-cmd-16130 [003] 158131.585284: sched_switch: prev_comm=trace-cmd prev_pid=16130 prev_prio=120 prev_state=S ==> next_comm=trace-cmd next_pid=16132 next_prio=120 Latency: 1155.677 usecs
+
+Average wakeup latency: 26626.656 usecs
+
+
+------------------------------------------
+
+The above trace produces the wakeup latencies of the tasks. The "sched_switch"
+event reports each individual latency after writing the event information.
+At the end of the report, the average wakeup latency is reported.
+
+------------------------------------------
+  # trace-cmd report -w -F 'sched_switch, sched_wakeup.*: prio < 100 || next_prio < 100'
+          <idle>-0     [003] 158131.516753: sched_wakeup: comm=ksoftirqd/3 pid=13 prio=49 success=1 target_cpu=003
+          <idle>-0     [003] 158131.516855: sched_switch: prev_comm=swapper prev_pid=0 prev_prio=120 prev_state=R ==> next_comm=ksoftirqd/3 next_pid=13 next_prio=49 Latency: 101.244 usecs
+          <idle>-0     [003] 158131.533781: sched_wakeup: comm=ksoftirqd/3 pid=13 prio=49 success=1 target_cpu=003
+          <idle>-0     [003] 158131.533897: sched_switch: prev_comm=swapper prev_pid=0 prev_prio=120 prev_state=R ==> next_comm=ksoftirqd/3 next_pid=13 next_prio=49 Latency: 115.608 usecs
+          <idle>-0     [003] 158131.569730: sched_wakeup: comm=ksoftirqd/3 pid=13 prio=49 success=1 target_cpu=003
+          <idle>-0     [003] 158131.569851: sched_switch: prev_comm=swapper prev_pid=0 prev_prio=120 prev_state=R ==> next_comm=ksoftirqd/3 next_pid=13 next_prio=49 Latency: 121.024 usecs
+
+Average wakeup latency: 110.021 usecs
+
+
+------------------------------------------
+
+The above version will only show the wakeups and context switches of Real Time
+tasks. The 'prio' used inside the kernel starts at 0 for highest priority.
+That is 'prio' 0 is equivalent to user space real time priority 99, and
+priority 98 is quivalent to user space real time priority 1.
+Prios less than 100 represent Real Time tasks.
+
+
+SEE ALSO
+--------
+trace-cmd(1), trace-cmd-record(1), trace-cmd-start(1), trace-cmd-stop(1),
+trace-cmd-extract(1), trace-cmd-reset(1), trace-cmd-split(1),
+trace-cmd-list(1), trace-cmd-listen(1)
+
+AUTHOR
+------
+Written by Steven Rostedt, <rostedt@goodmis.org>
+
+RESOURCES
+---------
+git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/trace-cmd.git
+
+COPYING
+-------
+Copyright \(C) 2010 Red Hat, Inc. Free use of this software is granted under
+the terms of the GNU Public License (GPL).
+
diff --git a/Documentation/trace-cmd-reset.txt b/Documentation/trace-cmd-reset.txt
new file mode 100644
index 0000000..0473e1d
--- /dev/null
+++ b/Documentation/trace-cmd-reset.txt
@@ -0,0 +1,54 @@
+TRACE-CMD-RESET(1)
+==================
+
+NAME
+----
+trace-cmd-reset - turn off all Ftrace tracing to bring back full performance
+
+SYNOPSIS
+--------
+*trace-cmd reset* ['OPTIONS']
+
+DESCRIPTION
+-----------
+The trace-cmd(1) reset command turns off all tracing of Ftrace. This will
+bring back the performance of the system before tracing was enabled. This is
+necessary since 'trace-cmd-record(1)', 'trace-cmd-stop(1)' and
+'trace-cmd-extract(1)' do not disable the tracer, event after the data has
+been pulled from the buffers. The rational is that the user may want to
+manually enable the tracer with the Ftrace pseudo file sytem, or examine other
+parts of Ftrace to see what trace-cmd did. After the reset command happens,
+the data in the ring buffer, and the options that were used are all lost.
+
+OPTIONS
+-------
+*-b* 'buffer_size'::
+    When the kernel boots, the Ftrace ring buffer is of a miminal size (3
+    pages per CPU). The first time the tracer is used, the ring buffer size
+    expands to what it was set for (default 1.4 Megs per CPU).
+
+    If no more tracing is to be done, this option allows you to shrink the
+    ring buffer down to free up available memory.
+
+    trace-cmd reset -b 1
+
+
+SEE ALSO
+--------
+trace-cmd(1), trace-cmd-record(1), trace-cmd-report(1), trace-cmd-start(1),
+trace-cmd-stop(1), trace-cmd-extract(1), trace-cmd-split(1),
+trace-cmd-list(1), trace-cmd-listen(1)
+
+AUTHOR
+------
+Written by Steven Rostedt, <rostedt@goodmis.org>
+
+RESOURCES
+---------
+git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/trace-cmd.git
+
+COPYING
+-------
+Copyright \(C) 2010 Red Hat, Inc. Free use of this software is granted under
+the terms of the GNU Public License (GPL).
+
diff --git a/Documentation/trace-cmd-split.txt b/Documentation/trace-cmd-split.txt
new file mode 100644
index 0000000..43d6df3
--- /dev/null
+++ b/Documentation/trace-cmd-split.txt
@@ -0,0 +1,98 @@
+TRACE-CMD-SPLIT(1)
+==================
+
+NAME
+----
+trace-cmd-split - split a trace.dat file into smaller files
+
+SYNOPSIS
+--------
+*trace-cmd split* ['OPTIONS'] ['start-time' ['end-time']]
+
+DESCRIPTION
+-----------
+The trace-cmd(1) split is used to break up a trace.dat into small files.
+The 'start-time' specifies where the new file will start at. Using
+'trace-cmd-report(1)' and copying the time stamp given at a particular event,
+can be used as input for either 'start-time' or 'end-time'. The split will
+stop creating files when it reaches an event after 'end-time'. If only the
+end-time is needed, use 0.0 as the start-time.
+
+If start-time is left out, then the split will start at the beginning of the
+file. If end-time is left out, then split will continue to the end unless it
+meets one of the reqirements specified by the options.
+
+OPTIONS
+-------
+*-i* 'file'::
+    If this option is not specified, then the split command will look for the
+    file named 'trace.dat'. This options will allow the reading of another
+    file other than 'trace.dat'.
+
+*-o* 'file'::
+    By default, the split command will use the input file name as a basis of
+    where to write the split files. The output file will be the input file
+    with an attached \'.#\' to the end: trace.dat.1, trace.dat.2, etc.
+
+    This option will change the name of the base file used.
+
+    -o file  will create file.1, file.2, etc.
+
+*-s* 'seconds'::
+    This specifies how many seconds should be recorded before the new file
+    should stop.
+
+*-m* 'milliseconds'::
+    This specifies how many milliseconds should be recorded before the new
+    file should stop.
+
+*-u* 'microseconds'::
+    This specifies how many microseconds should be recorded before the new
+    file should stop.
+
+*-e* 'events'::
+    This specifies how many events should be recorded before the new file
+    should stop.
+
+*-p* 'pages'::
+    This specifies the number of pages that should be recorded before the new
+    file should stop.
+
+   Note: only one of *-p*, *-e*, *-u*, *-m*, *-s* may be specified at a time.
+
+*-r*::
+    This option causes the break up to repeat until end-time is reached (or
+    end of the input if end-time is not specified).
+
+    trace-cmd split -r -e 10000
+
+    This will break up trace.dat into serval smaller files, each with at most
+    10,000 events in it.
+
+*-c*::
+    This option causes the above break up to be per CPU.
+
+    trace-cmd split -c -p 10
+
+    This will create a file that has 10 pages per each CPU from the input.
+
+
+SEE ALSO
+--------
+trace-cmd(1), trace-cmd-record(1), trace-cmd-report(1), trace-cmd-start(1),
+trace-cmd-stop(1), trace-cmd-extract(1), trace-cmd-reset(1),
+trace-cmd-list(1), trace-cmd-listen(1)
+
+AUTHOR
+------
+Written by Steven Rostedt, <rostedt@goodmis.org>
+
+RESOURCES
+---------
+git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/trace-cmd.git
+
+COPYING
+-------
+Copyright \(C) 2010 Red Hat, Inc. Free use of this software is granted under
+the terms of the GNU Public License (GPL).
+
diff --git a/Documentation/trace-cmd-start.txt b/Documentation/trace-cmd-start.txt
new file mode 100644
index 0000000..3264abf
--- /dev/null
+++ b/Documentation/trace-cmd-start.txt
@@ -0,0 +1,44 @@
+TRACE-CMD-START(1)
+===================
+
+NAME
+----
+trace-cmd-start - start the Ftrace Linux kernel tracer without recording
+
+SYNOPSIS
+--------
+*trace-cmd start* ['OPTIONS']
+
+DESCRIPTION
+-----------
+The trace-cmd(1) start enables all the Ftrace tracing the same way
+trace-cmd-record(1) does. The difference is that it does not run threads to
+create a trace.dat file. This is useful just to enable Ftrace and you are only
+interested in the trace after some event has occurred and the trace is
+stopped. Then the trace can be read straight from the Ftrace pseudo file
+system or can be extracted with trace-cmd-extract(1).
+
+OPTIONS
+-------
+The options are the same as 'trace-cmd-record(1)', except that it does not
+take options specific to recording (*-s*, *-o*, *-F*, *-N*, and *-t*).
+
+SEE ALSO
+--------
+trace-cmd(1), trace-cmd-record(1), trace-cmd-report(1), trace-cmd-stop(1),
+trace-cmd-extract(1), trace-cmd-reset(1), trace-cmd-split(1),
+trace-cmd-list(1), trace-cmd-listen(1)
+
+AUTHOR
+------
+Written by Steven Rostedt, <rostedt@goodmis.org>
+
+RESOURCES
+---------
+git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/trace-cmd.git
+
+COPYING
+-------
+Copyright \(C) 2010 Red Hat, Inc. Free use of this software is granted under
+the terms of the GNU Public License (GPL).
+
diff --git a/Documentation/trace-cmd-stop.txt b/Documentation/trace-cmd-stop.txt
new file mode 100644
index 0000000..c401af1
--- /dev/null
+++ b/Documentation/trace-cmd-stop.txt
@@ -0,0 +1,46 @@
+TRACE-CMD-STOP(1)
+===================
+
+NAME
+----
+trace-cmd-stop - stop the Ftrace Linux kernel tracer from writing to the ring
+buffer.
+
+SYNOPSIS
+--------
+*trace-cmd stop*
+
+DESCRIPTION
+-----------
+The trace-cmd(1) stop is a complement to 'trace-cmd-start(1)'. This will
+disable Ftrace from writing to the ring buffer. This does not stop the
+overhead that the tracing may incur. Only the updating of the ring buffer is
+disabled, the Ftrace tracing may still be inducing overhead.
+
+After stopping the trace, the 'trace-cmd-extract(1)' may strip out the data
+from the ring buffer and create a trace.dat file. The Ftrace pseudo file
+system may also be examined.
+
+To disable the tracing completely to remove the overhead it causes, use
+'trace-cmd-reset(1)'. But after a reset is performed, the data that has been
+recorded is lost.
+
+SEE ALSO
+--------
+trace-cmd(1), trace-cmd-record(1), trace-cmd-report(1), trace-cmd-start(1),
+trace-cmd-extract(1), trace-cmd-reset(1), trace-cmd-split(1),
+trace-cmd-list(1), trace-cmd-listen(1)
+
+AUTHOR
+------
+Written by Steven Rostedt, <rostedt@goodmis.org>
+
+RESOURCES
+---------
+git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/trace-cmd.git
+
+COPYING
+-------
+Copyright \(C) 2010 Red Hat, Inc. Free use of this software is granted under
+the terms of the GNU Public License (GPL).
+
diff --git a/Documentation/trace-cmd.txt b/Documentation/trace-cmd.txt
new file mode 100644
index 0000000..6a02cbe
--- /dev/null
+++ b/Documentation/trace-cmd.txt
@@ -0,0 +1,73 @@
+TRACE-CMD(1)
+============
+
+NAME
+----
+trace-cmd - interacts with Ftrace Linux kernel internal tracer
+
+SYNOPSIS
+--------
+*trace-cmd* 'COMMAND' ['OPTIONS']
+
+DESCRIPTION
+-----------
+The trace-cmd(1) command interacts with the Ftrace tracer that is built inside
+the Linux kernel. It intefaces with the Ftrace specific files found in the
+debugfs file system under the tracing directory. A 'COMMAND' must be
+specified to tell trace-cmd what to do.
+
+
+COMMANDS
+--------
+
+  record  - record a live trace and write a trace.dat file to the
+            local disk or to the network.
+
+  report  - reads a trace.dat file and converts the binary data to a
+            ASCII text readable format.
+
+  start   - start the tracing without recording to a trace.dat file.
+
+  stop    - stop tracing (only disables recording, overhead of tracer
+            is still in effect)
+
+  extract - extract the data from the kernel buffer and create a trace.dat
+            file.
+
+  reset   - disables all tracing and gives back the system performance.
+            (clears all data from the kernel buffers)
+
+  split   - splits a trace.dat file into smaller files.
+
+  list    - list the available plugins or events that can be recorded.
+
+  listen  - open up a port to listen for remote tracing connections.
+
+
+OPTIONS
+-------
+
+*-h, --help::
+    Displace the help text.
+
+Other options see the man page for the corresponding command.
+
+SEE ALSO
+--------
+trace-cmd-record(1), trace-cmd-report(1), trace-cmd-start(1),
+trace-cmd-stop(1), trace-cmd-extract(1), trace-cmd-reset(1),
+trace-cmd-split(1), trace-cmd-list(1), trace-cmd-listen(1)
+
+AUTHOR
+------
+Written by Steven Rostedt, <rostedt@goodmis.org>
+
+RESOURCES
+---------
+git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/trace-cmd.git
+
+COPYING
+-------
+Copyright \(C) 2010 Red Hat, Inc. Free use of this software is granted under
+the terms of the GNU Public License (GPL).
+