aboutsummaryrefslogtreecommitdiffstats
path: root/tools/perf/Documentation
Commit message (Collapse)AuthorAge
* perf trace: update Documentation with new perf trace variantsTom Zanussi2010-11-10
| | | | | | | | Add documentation describing new 'perf trace' command changes e.g. <command> handling and live-mode/top variants. Signed-off-by: Tom Zanussi <tom.zanussi@linux.intel.com> Acked-by: Arnaldo Carvalho de Melo <acme@ghostprotocols.net>
* perf scripting: Shut up 'perf record' final statusArnaldo Carvalho de Melo2010-10-26
| | | | | | | | | | | | | | We want just the script output, not internal details about the record phase. Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: Ingo Molnar <mingo@elte.hu> Cc: Mike Galbraith <efault@gmx.de> Cc: Paul Mackerras <paulus@samba.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Stephane Eranian <eranian@google.com> Cc: Tom Zanussi <tzanussi@gmail.com> LKML-Reference: <new-submission> Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
* perf tools: Document event modifiersSonny Rao2010-10-23
| | | | | | | | | | | | | Existing documentation doesn't discuss event modifiers, so add a description of what's currently possible to the documentation of perf-list. Cc: Ingo Molnar <mingo@elte.hu> Cc: Paul Mackerras <paulus@samba.org> Cc: Peter Zijlstra <a.p.zijlstra@chello.nl> Cc: Robert Richter <robert.richter@amd.com> LKML-Reference: <1287107460-12112-1-git-send-email-sonnyrao@linux.vnet.ibm.com> Signed-off-by: Sonny Rao <sonnyrao@linux.vnet.ibm.com> Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
* perf probe: Add basic module supportMasami Hiramatsu2010-10-21
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Add basic module probe support on perf probe. This introduces "--module <MODNAME>" option to perf probe for putting probes and showing lines and variables in the given module. Currently, this supports only probing on running modules. Supporting off-line module probing is the next step. e.g.) [show lines] # ./perf probe --module drm -L drm_vblank_info <drm_vblank_info:0> 0 int drm_vblank_info(struct seq_file *m, void *data) 1 { struct drm_info_node *node = (struct drm_info_node *) m->private 3 struct drm_device *dev = node->minor->dev; ... [show vars] # ./perf probe --module drm -V drm_vblank_info:3 Available variables at drm_vblank_info:3 @<drm_vblank_info+20> (unknown_type) data struct drm_info_node* node struct seq_file* m [put a probe] # ./perf probe --module drm drm_vblank_info:3 node m Add new event: probe:drm_vblank_info (on drm_vblank_info:3 with node m) You can now use it on all perf tools, such as: perf record -e probe:drm_vblank_info -aR sleep 1 [list probes] # ./perf probe -l probe:drm_vblank_info (on drm_vblank_info:3@drivers/gpu/drm/drm_info.c with ... Cc: Peter Zijlstra <a.p.zijlstra@chello.nl> Cc: Paul Mackerras <paulus@samba.org> Cc: Ingo Molnar <mingo@elte.hu> Cc: Frederic Weisbecker <fweisbec@gmail.com> LKML-Reference: <20101021101341.3542.71638.stgit@ltc236.sdl.hitachi.co.jp> Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
* perf probe: Show accessible global variablesMasami Hiramatsu2010-10-21
| | | | | | | | | | | | | | | Add --externs for allowing --vars to show accessible global (externally defined) variables from a given probe point too. This will give you a hint which globals can be accessible from the probe point. Cc: Peter Zijlstra <a.p.zijlstra@chello.nl> Cc: Paul Mackerras <paulus@samba.org> Cc: Ingo Molnar <mingo@elte.hu> Cc: Frederic Weisbecker <fweisbec@gmail.com> LKML-Reference: <20101021101335.3542.31003.stgit@ltc236.sdl.hitachi.co.jp> Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
* perf probe: Show accessible local variablesMasami Hiramatsu2010-10-21
| | | | | | | | | | | | | | | | | | | | | | | | | | Add -V (--vars) option for listing accessible local variables at given probe point. This will help finding which local variables are available for event arguments. e.g.) # perf probe -V call_timer_fn:23 Available variables at call_timer_fn:23 @<run_timer_softirq+345> function_type* fn int preempt_count long unsigned int data struct list_head work_list struct list_head* head struct timer_list* timer struct tvec_base* base Cc: Peter Zijlstra <a.p.zijlstra@chello.nl> Cc: Paul Mackerras <paulus@samba.org> Cc: Ingo Molnar <mingo@elte.hu> Cc: Frederic Weisbecker <fweisbec@gmail.com> LKML-Reference: <20101021101323.3542.40282.stgit@ltc236.sdl.hitachi.co.jp> Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
* perf tools: Add --tui and --stdio to choose the UIArnaldo Carvalho de Melo2010-08-21
| | | | | | | | | | | | | | | Relying just on ~/.perfconfig or rebuilding the tool disabling support for the TUI is too cumbersome, so allow specifying which UI to use and make the command line switch override whatever is in ~/.perfconfig. Suggested-by: Christoph Hellwig <hch@infradead.org> Cc: Christoph Hellwig <hch@infradead.org> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: Mike Galbraith <efault@gmx.de> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Stephane Eranian <eranian@google.com> LKML-Reference: <new-submission> Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
* perf man pages: Fix cut'n'paste errorArnaldo Carvalho de Melo2010-07-28
| | | | | | | | | | | We remove files _from_ the cache. Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: Mike Galbraith <efault@gmx.de> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Stephane Eranian <eranian@google.com> LKML-Reference: <new-submission> Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
* perf probe: Support tracing an entry of arrayMasami Hiramatsu2010-07-05
| | | | | | | | | | | | | | | | | | Add array-entry tracing support to perf probe. This enables to trace an entry of array which is indexed by constant value, e.g. array[0]. For example: $ perf probe -a 'bio_split bi->bi_io_vec[0]' Cc: Paul Mackerras <paulus@samba.org> Cc: Ingo Molnar <mingo@elte.hu> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Mike Galbraith <efault@gmx.de> Cc: Frederic Weisbecker <fweisbec@gmail.com> LKML-Reference: <20100519195742.2885.5344.stgit@localhost6.localdomain6> Signed-off-by: Masami Hiramatsu <mhiramat@redhat.com> Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
* perf probe: Support "string" typeMasami Hiramatsu2010-07-05
| | | | | | | | | | | | | | | | | Support string type casting to event argument. If perf-probe finds an argument casted as string, it ensures the target variable is "(unsigned/signed) char *(or []). perf-probe also adds dereference if the target is a pointer. So, both of 'char buf[10];' and 'char *buf;' can be accessed by 'buf:string' Cc: Ingo Molnar <mingo@elte.hu> Cc: Paul Mackerras <paulus@samba.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Mike Galbraith <efault@gmx.de> Cc: Frederic Weisbecker <fweisbec@gmail.com> LKML-Reference: <20100519195734.2885.1666.stgit@localhost6.localdomain6> Signed-off-by: Masami Hiramatsu <mhiramat@redhat.com> Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
* perf probe: Add kernel source path optionChase Douglas2010-06-17
| | | | | | | | | | | | | | | | The probe plugin requires access to the source code for some operations. The source code must be in the exact same location as specified by the DWARF tags, but sometimes the location is an absolute path that cannot be replicated by a normal user. This change adds the -s|--source option to allow the user to specify the root of the kernel source tree. Cc: Ingo Molnar <mingo@elte.hu> Cc: Paul Mackerras <paulus@samba.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Masami Hiramatsu <mhiramat@redhat.com> LKML-Reference: <1276543590-10486-1-git-send-email-chase.douglas@canonical.com> Signed-off-by: Chase Douglas <chase.douglas@canonical.com> Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
* perf record: Add option to avoid updating buildid cacheStephane Eranian2010-06-17
| | | | | | | | | | | | | | | | | | | | | | | | | There are situations where there is enough information in the perf.data to process the samples. Updating the buildid cache may add unecessary overhead in terms of disk space and time (copying large elf images). A persistent option to do this already exists via the perfconfig file, simply do: [buildid] dir = /dev/null This patch provides a way to suppress builid cache updates on a per-run basis. It addds a new option, -N, to perf record. Buildids are still generated in the perf.data file. Cc: David S. Miller <davem@davemloft.net> Cc: Frédéric Weisbecker <fweisbec@gmail.com> Cc: Ingo Molnar <mingo@elte.hu> Cc: Paul Mackerras <paulus@samba.org> Cc: Peter Zijlstra <peterz@infradead.org> LKML-Reference: <4c19ef89.93ecd80a.40dc.fffff8e9@mx.google.com> Signed-off-by: Stephane Eranian <eranian@google.com> Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
* perf tools: Add the ability to specify list of cpus to monitorStephane Eranian2010-06-05
| | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch adds a -C option to stat, record, top to designate a list of CPUs to monitor. CPUs can be specified as a comma-separated list or ranges, no space allowed. Examples: $ perf record -a -C0-1,4-7 sleep 1 $ perf top -C0-4 $ perf stat -a -C1,2,3,4 sleep 1 With perf record in per-thread mode with inherit mode on, samples are collected only when the thread runs on the designated CPUs. The -C option does not turn on system-wide mode automatically. Cc: David S. Miller <davem@davemloft.net> Cc: Frédéric Weisbecker <fweisbec@gmail.com> Cc: Ingo Molnar <mingo@elte.hu> Cc: Mike Galbraith <efault@gmx.de> Cc: Paul Mackerras <paulus@samba.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Stephane Eranian <eranian@google.com> Cc: Tom Zanussi <tzanussi@gmail.com> LKML-Reference: <4bff9496.d345d80a.41fe.7b00@mx.google.com> Signed-off-by: Stephane Eranian <eranian@google.com> Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
* perf stat: add perf stat -B to pretty print large numbersStephane Eranian2010-05-18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | It is hard to read very large numbers so provide an option to perf stat to separate thousands using a separator. The patch leverages the locale support of stdio. You need to set your LC_NUMERIC appropriately, for instance LC_NUMERIC=en_US.UTF8. You need to pass -B to activate this feature. This way existing scripts parsing the output do not need to be changed. Here is an example. $ perf stat noploop 2 noploop for 2 seconds Performance counter stats for 'noploop 2': 1998.347031 task-clock-msecs # 0.998 CPUs 61 context-switches # 0.000 M/sec 0 CPU-migrations # 0.000 M/sec 118 page-faults # 0.000 M/sec 4,138,410,900 cycles # 2070.917 M/sec (scaled from 70.01%) 2,062,650,268 instructions # 0.498 IPC (scaled from 70.01%) 2,057,653,466 branches # 1029.678 M/sec (scaled from 70.01%) 40,267 branch-misses # 0.002 % (scaled from 30.04%) 2,055,961,348 cache-references # 1028.831 M/sec (scaled from 30.03%) 53,725 cache-misses # 0.027 M/sec (scaled from 30.02%) 2.001393933 seconds time elapsed $ perf stat -B noploop 2 noploop for 2 seconds Performance counter stats for 'noploop 2': 1998.297883 task-clock-msecs # 0.998 CPUs 59 context-switches # 0.000 M/sec 0 CPU-migrations # 0.000 M/sec 119 page-faults # 0.000 M/sec 4,131,380,160 cycles # 2067.450 M/sec (scaled from 70.01%) 2,059,096,507 instructions # 0.498 IPC (scaled from 70.01%) 2,054,681,303 branches # 1028.216 M/sec (scaled from 70.01%) 25,650 branch-misses # 0.001 % (scaled from 30.05%) 2,056,283,014 cache-references # 1029.017 M/sec (scaled from 30.03%) 47,097 cache-misses # 0.024 M/sec (scaled from 30.02%) 2.001391016 seconds time elapsed Cc: David S. Miller <davem@davemloft.net> Cc: Frédéric Weisbecker <fweisbec@gmail.com> Cc: Ingo Molnar <mingo@elte.hu> Cc: Mike Galbraith <efault@gmx.de> Cc: Paul Mackerras <paulus@samba.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Tom Zanussi <tzanussi@gmail.com> LKML-Reference: <4bf28fe8.914ed80a.01ca.fffff5f5@mx.google.com> Signed-off-by: Stephane Eranian <eranian@google.com> Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
* perf trace scripts: Fix typos in perf-trace-python.txtKirill Smelkov2010-05-13
| | | | | | | | | | option option -> option special special -> special Signed-off-by: Kirill Smelkov <kirr@mns.spb.ru> Cc: Tom Zanussi <tzanussi@gmail.com> LKML-Reference: <1273747165-17242-1-git-send-email-kirr@mns.spb.ru> Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
* perf tools: change event inheritance logic in stat and recordStephane Eranian2010-05-13
| | | | | | | | | | | | | | | | | | | | | | | | By default, event inheritance across fork and pthread_create was on but the -i option of stat and record, which enabled inheritance, led to believe it was off by default. This patch fixes this logic by inverting the meaning of the -i option. By default inheritance is on whether you attach to a process (-p), a thread (-t) or start a process. If you pass -i, then you turn off inheritance. Turning off inheritance if you don't need it, helps limit perf resource usage as well. The patch also fixes perf stat -t xxxx and perf record -t xxxx which did not start the counters. Acked-by: Frederic Weisbecker <fweisbec@gmail.com> Cc: David S. Miller <davem@davemloft.net> Cc: Frédéric Weisbecker <fweisbec@gmail.com> Cc: Ingo Molnar <mingo@elte.hu> Cc: Paul Mackerras <paulus@samba.org> Cc: Peter Zijlstra <peterz@infradead.org> LKML-Reference: <4bea9d2f.d60ce30a.0b5b.08e1@mx.google.com> Signed-off-by: Stephane Eranian <eranian@google.com> Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
* perf list: Improve the raw hw event descriptor documentationArnaldo Carvalho de Melo2010-05-07
| | | | | | | | | | | | | | | | | | It was x86 specific and imcomplete at that, improve the situation by making it clear where the example provided applies and by adding the URLs for the Intel and AMD manuals where this is discussed in depth. Acked-by: Robert Richter <robert.richter@amd.com> Cc: Cyrill Gorcunov <gorcunov@gmail.com> Cc: Frédéric Weisbecker <fweisbec@gmail.com> Cc: Mike Galbraith <efault@gmx.de> Cc: Paul Mackerras <paulus@samba.org> Cc: Peter Zijlstra <a.p.zijlstra@chello.nl> Cc: Tom Zanussi <tzanussi@gmail.com> Cc: Robert Richter <robert.richter@amd.com> Reported-by: Robert Richter <robert.richter@amd.com LKML-Reference: <new-submission> Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
* perf tools: Fixup minor doc formatting issuesArnaldo Carvalho de Melo2010-05-05
| | | | | | | | | | Cc: Frédéric Weisbecker <fweisbec@gmail.com> Cc: Mike Galbraith <efault@gmx.de> Cc: Paul Mackerras <paulus@samba.org> Cc: Peter Zijlstra <a.p.zijlstra@chello.nl> Cc: Tom Zanussi <tzanussi@gmail.com> LKML-Reference: <new-submission> Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
* perf list: Add explanation about raw hardware event descriptorsArnaldo Carvalho de Melo2010-05-05
| | | | | | | | | | | | | | Using explanation given by Ingo Molnar in the oprofile mailing list. Suggested-by: Nick Black <dank@qemfd.net> Cc: Frédéric Weisbecker <fweisbec@gmail.com> Cc: Mike Galbraith <efault@gmx.de> Cc: Nick Black <dank@qemfd.net> Cc: Paul Mackerras <paulus@samba.org> Cc: Peter Zijlstra <a.p.zijlstra@chello.nl> Cc: Tom Zanussi <tzanussi@gmail.com> LKML-Reference: <new-submission> Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
* Merge branch 'perf' of ↵Ingo Molnar2010-05-04
|\ | | | | | | git://git.kernel.org/pub/scm/linux/kernel/git/acme/linux-2.6 into perf/core
| * perf inject: Add missing bitsArnaldo Carvalho de Melo2010-05-04
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | New commands need to have Documentation and be added to command-list.txt so that they can appear when 'perf' is called withouth any subcommand: [root@doppio linux-2.6-tip]# perf usage: perf [--version] [--help] COMMAND [ARGS] The most commonly used perf commands are: annotate Read perf.data (created by perf record) and display annotated code archive Create archive with object files with build-ids found in perf.data file bench General framework for benchmark suites buildid-cache Manage build-id cache. buildid-list List the buildids in a perf.data file diff Read two perf.data files and display the differential profile inject Filter to augment the events stream with additional information kmem Tool to trace/measure kernel memory(slab) properties kvm Tool to trace/measure kvm guest os list List all symbolic event types lock Analyze lock events probe Define new dynamic tracepoints record Run a command and record its profile into perf.data report Read perf.data (created by perf record) and display the profile sched Tool to trace/measure scheduler properties (latencies) stat Run a command and gather performance counter statistics test Runs sanity tests. timechart Tool to visualize total system behavior during a workload top System profiling tool. trace Read perf.data (created by perf record) and display trace output See 'perf help COMMAND' for more information on a specific command. [root@doppio linux-2.6-tip]# The new 'perf inject' command hadn't so it wasn't appearing on that list. Also fix the long option, that should have no spaces in it, rename the faulty one to be '--build-ids', instead of '--inject build-ids'. Reported-by: Ingo Molnar <mingo@elte.hu> Cc: Frédéric Weisbecker <fweisbec@gmail.com> Cc: Mike Galbraith <efault@gmx.de> Cc: Paul Mackerras <paulus@samba.org> Cc: Peter Zijlstra <a.p.zijlstra@chello.nl> Cc: Tom Zanussi <tzanussi@gmail.com> LKML-Reference: <new-submission> Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
* | perf: Remove leftover useless options to record trace events from scriptsFrederic Weisbecker2010-04-30
|/ | | | | | | | | | | | | | -f, -c 1, -R are now useless for trace events recording, moreover -M is useless and event hurts. Remove them from the documentation examples and from record scripts. Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com> Cc: Ingo Molnar <mingo@elte.hu> Cc: Peter Zijlstra <a.p.zijlstra@chello.nl> Cc: Arnaldo Carvalho de Melo <acme@redhat.com> Cc: Paul Mackerras <paulus@samba.org> Cc: Tom Zanussi <tzanussi@gmail.com>
* perf test: Initial regression testing commandArnaldo Carvalho de Melo2010-04-29
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | First an example with the first internal test: [acme@doppio linux-2.6-tip]$ perf test 1: vmlinux symtab matches kallsyms: Ok So it run just one test, that is "vmlinux symtab matches kallsyms", and it was successful. If we run it in verbose mode, we'll see details about errors and extra warnings for non-fatal problems: [acme@doppio linux-2.6-tip]$ perf test -v 1: vmlinux symtab matches kallsyms: --- start --- Looking at the vmlinux_path (5 entries long) No build_id in vmlinux, ignoring it No build_id in /boot/vmlinux, ignoring it No build_id in /boot/vmlinux-2.6.34-rc4-tip+, ignoring it Using /lib/modules/2.6.34-rc4-tip+/build/vmlinux for symbols Maps only in vmlinux: ffffffff81cb81b1-ffffffff81e1149b 0 [kernel].init.text ffffffff81e1149c-ffffffff9fffffff 0 [kernel].exit.text ffffffffff600000-ffffffffff6000ff 0 [kernel].vsyscall_0 ffffffffff600100-ffffffffff6003ff 0 [kernel].vsyscall_fn ffffffffff600400-ffffffffff6007ff 0 [kernel].vsyscall_1 ffffffffff600800-ffffffffffffffff 0 [kernel].vsyscall_2 Maps in vmlinux with a different name in kallsyms: ffffffffff600000-ffffffffff6000ff 0 [kernel].vsyscall_0 in kallsyms as [kernel].0 ffffffffff600100-ffffffffff6003ff 0 [kernel].vsyscall_fn in kallsyms as: *ffffffffff600100-ffffffffff60012f 0 [kernel].2 ffffffffff600400-ffffffffff6007ff 0 [kernel].vsyscall_1 in kallsyms as [kernel].6 ffffffffff600800-ffffffffffffffff 0 [kernel].vsyscall_2 in kallsyms as [kernel].8 Maps only in kallsyms: ffffffffff600130-ffffffffff6003ff 0 [kernel].4 ---- end ---- vmlinux symtab matches kallsyms: Ok [acme@doppio linux-2.6-tip]$ In the above case we only know the name of the non contiguous kernel ranges in the address space when reading the symbol information from the ELF symtab in vmlinux. The /proc/kallsyms file lack this, we only notice they are separate because there are modules after the kernel and after that more kernel functions, so we need to have a module rbtree backed by the module .ko path to get symtabs in the vmlinux case. The tool uses it to match by address to emit appropriate warning, but don't considers this fatal. The .init.text and .exit.text ines, of course, aren't in kallsyms, so I left these cases just as extra info in verbose mode. The end of the sections also aren't in kallsyms, so we the symbols layer does another pass and sets the end addresses as the next map start minus one, which sometimes pads, causing harmless mismatches. But at least the symbols match, tested it by copying /proc/kallsyms to /tmp/kallsyms and doing changes to see if they were detected. This first test also should serve as a first stab at documenting the symbol library by providing a self contained example that exercises it together with comments about what is being done. More tests to check if actions done on a monitored app, like doing mmaps, etc, makes the kernel generate the expected events should be added next. Cc: Frédéric Weisbecker <fweisbec@gmail.com> Cc: Mike Galbraith <efault@gmx.de> Cc: Paul Mackerras <paulus@samba.org> Cc: Peter Zijlstra <a.p.zijlstra@chello.nl> LKML-Reference: <new-submission> Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
* Merge branch 'perf' of ↵Ingo Molnar2010-04-27
|\ | | | | | | git://git.kernel.org/pub/scm/linux/kernel/git/acme/linux-2.6 into perf/core
| * perf probe: Add --max-probes optionMasami Hiramatsu2010-04-26
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Add --max-probes option to change the maximum limit of findable probe points per event, since inlined function can be expanded into thousands of probe points. Default value is 128. Signed-off-by: Masami Hiramatsu <mhiramat@redhat.com> Suggested-by: Arnaldo Carvalho de Melo <acme@redhat.com> Cc: Paul Mackerras <paulus@samba.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Mike Galbraith <efault@gmx.de> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: Ingo Molnar <mingo@elte.hu> LKML-Reference: <20100421195640.24664.62984.stgit@localhost6.localdomain6> Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
* | perf: Some perf-kvm documentation editsWilliam Cohen2010-04-23
|/ | | | | | | | | | | | | | asciidoc does not allow the "===" to be longer than the line above it. Also fix a couple types and formatting errors. Signed-off-by: William Cohen <wcohen@redhat.com> Cc: Peter Zijlstra <a.p.zijlstra@chello.nl> Cc: Arnaldo Carvalho de Melo <acme@redhat.com> Cc: Paul Mackerras <paulus@samba.org> Cc: Ingo Molnar <mingo@elte.hu> LKML-Reference: <4BD204C5.9000504@redhat.com> Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
* perf: 'perf kvm' tool for monitoring guest performance from hostZhang, Yanmin2010-04-19
| | | | | | | Here is the patch of userspace perf tool. Signed-off-by: Zhang Yanmin <yanmin_zhang@linux.intel.com> Signed-off-by: Avi Kivity <avi@redhat.com>
* Merge branch 'perf' of ↵Ingo Molnar2010-04-15
|\ | | | | | | git://git.kernel.org/pub/scm/linux/kernel/git/acme/linux-2.6 into perf/core
| * perf probe: Support basic type castingMasami Hiramatsu2010-04-14
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Add basic type casting for arguments to perf probe. This allows users to specify the actual type of arguments. Of course, if user sets invalid types, kprobe-tracer rejects that. Cc: Ingo Molnar <mingo@elte.hu> Cc: Paul Mackerras <paulus@samba.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Mike Galbraith <efault@gmx.de> Cc: Frederic Weisbecker <fweisbec@gmail.com> LKML-Reference: <20100412171722.3790.50372.stgit@localhost6.localdomain6> Signed-off-by: Masami Hiramatsu <mhiramat@redhat.com> Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
| * perf probe: Use the last field name as the argument nameMasami Hiramatsu2010-04-14
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Set the last field name to the argument name when the argument is refering a data-structure member. e.g. ./perf probe --add 'vfs_read file->f_mode' Add new event: probe:vfs_read (on vfs_read with f_mode=file->f_mode) This probe records file->f_mode, but the argument name becomes "f_mode". This enables perf-trace command to parse trace event format correctly. Cc: Ingo Molnar <mingo@elte.hu> Cc: Paul Mackerras <paulus@samba.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Mike Galbraith <efault@gmx.de> Cc: Frederic Weisbecker <fweisbec@gmail.com> LKML-Reference: <20100412171700.3790.72961.stgit@localhost6.localdomain6> Signed-off-by: Masami Hiramatsu <mhiramat@redhat.com> Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
| * perf probe: Support argument nameMasami Hiramatsu2010-04-14
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Set given names to event arguments. The syntax is same as kprobe-tracer, you can add 'NAME=' right before each argument. e.g. ./perf probe vfs_read foo=file then, 'foo' is set to the argument name as below. ./perf probe -l probe:vfs_read (on vfs_read@linux-2.6-tip/fs/read_write.c with foo) Cc: Ingo Molnar <mingo@elte.hu> Cc: Paul Mackerras <paulus@samba.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Mike Galbraith <efault@gmx.de> Cc: Frederic Weisbecker <fweisbec@gmail.com> LKML-Reference: <20100412171653.3790.74624.stgit@localhost6.localdomain6> Signed-off-by: Masami Hiramatsu <mhiramat@redhat.com> Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
* | perf: Always record tracepoints raw samples from perf recordFrederic Weisbecker2010-04-14
| | | | | | | | | | | | | | | | | | | | | | | | | | Trace events are mostly used for tracing rather than simple counting. Don't bother anymore with adding -R when using them, just record raw samples of trace events every time. Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com> Cc: Arnaldo Carvalho de Melo <acme@redhat.com> Cc: Peter Zijlstra <a.p.zijlstra@chello.nl> Cc: Paul Mackerras <paulus@samba.org> Cc: Ingo Molnar <mingo@elte.hu> Cc: Thomas Gleixner <tglx@linutronix.de>
* | perf: Make -f the default for perf recordFrederic Weisbecker2010-04-14
|/ | | | | | | | | | | | | | | | Force the overwriting mode by default if append mode is not explicit. Adding -f every time one uses perf on a daily basis quickly becomes a burden. Keep the -f among the options though to avoid breaking some random users scripts. Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com> Cc: Arnaldo Carvalho de Melo <acme@redhat.com> Cc: Peter Zijlstra <a.p.zijlstra@chello.nl> Cc: Paul Mackerras <paulus@samba.org> Cc: Ingo Molnar <mingo@elte.hu> Cc: Thomas Gleixner <tglx@linutronix.de>
* perf: cleanup some DocumentationRandy Dunlap2010-04-08
| | | | | | | | | Correct typos in perf bench & perf sched help text. Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>, Cc: Paul Mackerras <paulus@samba.org> LKML-Reference: <20100331113100.cc898487.randy.dunlap@oracle.com> Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>
* perf probe: Fix --line syntax help and documentMasami Hiramatsu2010-04-02
| | | | | | | | | | | | | | | | Just fix typos. --line option accepts ':START-END' syntax, not ':START:END'. Signed-off-by: Masami Hiramatsu <mhiramat@redhat.com> Cc: Paul Mackerras <paulus@samba.org> Cc: Arnaldo Carvalho de Melo <acme@redhat.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Mike Galbraith <efault@gmx.de> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: systemtap <systemtap@sources.redhat.com> Cc: DLE <dle-develop@lists.sourceforge.net> LKML-Reference: <20100402165038.23551.62590.stgit@localhost6.localdomain6> Signed-off-by: Ingo Molnar <mingo@elte.hu>
* perf probe: Add --dry-run optionMasami Hiramatsu2010-03-17
| | | | | | | | | | | | | | | Add --dry-run option for debugging and testing. Signed-off-by: Masami Hiramatsu <mhiramat@redhat.com> Cc: systemtap <systemtap@sources.redhat.com> Cc: DLE <dle-develop@lists.sourceforge.net> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: Arnaldo Carvalho de Melo <acme@redhat.com> Cc: Paul Mackerras <paulus@samba.org> Cc: Mike Galbraith <efault@gmx.de> Cc: Peter Zijlstra <a.p.zijlstra@chello.nl> LKML-Reference: <20100316220605.32050.6571.stgit@localhost6.localdomain6> Signed-off-by: Ingo Molnar <mingo@elte.hu>
* perf: Make the install relative to DESTDIR if specifiedJohn Kacur2010-03-11
| | | | | | | | | | | | | | | | | | | | | | | Without this change, the install path is relative to prefix/DESTDIR where prefix is automatically set to $HOME. This can produce unexpected results. For example: make -C tools/perf DESTDIR=/home/jkacur/tmp install-man creates the directory: /home/jkacur/home/jkacur/tmp/share/... instead of the expected: /home/jkacur/tmp/share/... Signed-off-by: John Kacur <jkacur@redhat.com> Cc: Peter Zijlstra <a.p.zijlstra@chello.nl> Cc: Paul Mackerras <paulus@samba.org> Cc: Arnaldo Carvalho de Melo <acme@redhat.com> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: Tom Zanussi <tzanussi@gmail.com> Cc: Kyle McMartin <kyle@redhat.com> Cc: <stable@kernel.org> LKML-Reference: <1268312220-12880-1-git-send-email-jkacur@redhat.com> Signed-off-by: Ingo Molnar <mingo@elte.hu>
* Merge commit 'v2.6.34-rc1' into perf/urgentIngo Molnar2010-03-09
|\ | | | | | | | | | | | | | | | | Conflicts: tools/perf/util/probe-event.c Merge reason: Pick up -rc1 and resolve the conflict as well. Signed-off-by: Ingo Molnar <mingo@elte.hu>
| * perf probe: Add lazy line matching supportMasami Hiramatsu2010-02-25
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Add lazy line matching support for specifying new probes. This also changes the syntax of perf probe a bit. Now perf probe accepts one of below probe event definitions. 1) Define event based on function name [EVENT=]FUNC[@SRC][:RLN|+OFF|%return|;PTN] [ARG ...] 2) Define event based on source file with line number [EVENT=]SRC:ALN [ARG ...] 3) Define event based on source file with lazy pattern [EVENT=]SRC;PTN [ARG ...] - New lazy matching pattern(PTN) follows ';' (semicolon). And it must be put the end of the definition. - So, @SRC is no longer the part which must be put at the end of the definition. Note that ';' (semicolon) can be interpreted as the end of a command by the shell. This means that you need to quote it. (anyway you will need to quote the lazy pattern itself too, because it may contains other sensitive characters, like '[',']' etc.). Lazy matching ------------- The lazy line matching is similar to glob matching except ignoring spaces in both of pattern and target. e.g. 'a=*' can matches 'a=b', 'a = b', 'a == b' and so on. This provides some sort of flexibility and robustness to probe point definitions against minor code changes. (for example, actual 10th line of schedule() can be changed easily by modifying schedule(), but the same line matching 'rq=cpu_rq*' may still exist.) Changes in v3: - Cast Dwarf_Addr to uintmax_t for printf-formats. Changes in v2: - Cast Dwarf_Addr to unsigned long long for printf-formats. Signed-off-by: Masami Hiramatsu <mhiramat@redhat.com> Cc: systemtap <systemtap@sources.redhat.com> Cc: DLE <dle-develop@lists.sourceforge.net> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: Arnaldo Carvalho de Melo <acme@redhat.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Paul Mackerras <paulus@samba.org> Cc: Mike Galbraith <efault@gmx.de> Cc: K.Prasad <prasad@linux.vnet.ibm.com> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: Ananth N Mavinakayanahalli <ananth@in.ibm.com> LKML-Reference: <20100225133611.6725.45078.stgit@localhost6.localdomain6> Signed-off-by: Ingo Molnar <mingo@elte.hu>
| * perf probe: Update perf probe documentMasami Hiramatsu2010-02-25
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Update perf-probe.txt to suit to current perf-probe command and add some examples. Signed-off-by: Masami Hiramatsu <mhiramat@redhat.com> Cc: systemtap <systemtap@sources.redhat.com> Cc: DLE <dle-develop@lists.sourceforge.net> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: Arnaldo Carvalho de Melo <acme@redhat.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Paul Mackerras <paulus@samba.org> Cc: Mike Galbraith <efault@gmx.de> Cc: K.Prasad <prasad@linux.vnet.ibm.com> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: Ananth N Mavinakayanahalli <ananth@in.ibm.com> LKML-Reference: <20100225133519.6725.58816.stgit@localhost6.localdomain6> Signed-off-by: Ingo Molnar <mingo@elte.hu>
* | perf lock: Fix and add misc documentally thingsHitoshi Mitake2010-02-27
|/ | | | | | | | | | | | | | | | | | | | I've forgot to add 'perf lock' line to command-list.txt, so users of perf could not find perf lock when they type 'perf'. Fixing command-list.txt requires document (tools/perf/Documentation/perf-lock.txt). But perf lock is too much "under construction" to write a stable document, so this is something like pseudo document for now. And I wrote description of perf lock at help section of CONFIG_LOCK_STAT, this will navigate users of lock trace events. Signed-off-by: Hitoshi Mitake <mitake@dcl.info.waseda.ac.jp> Cc: Peter Zijlstra <a.p.zijlstra@chello.nl> Cc: Paul Mackerras <paulus@samba.org> Cc: Arnaldo Carvalho de Melo <acme@ghostprotocols.net> LKML-Reference: <1265267295-8388-1-git-send-email-mitake@dcl.info.waseda.ac.jp> Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
* perf/scripts: Tag syscall_name helper as not yet availableFrederic Weisbecker2010-02-24
| | | | | | | | | | | | | | | | | | | syscall_name() helper, which resolves a syscall arch number to its name, is not yet available as we first need to implement event injection for it to work. Remove it from the documentation or tag its references as unavailable yet. Once it's implemented, we can just revert the current patch. Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com> Cc: Tom Zanussi <tzanussi@gmail.com> Cc: Ingo Molnar <mingo@elte.hu> Cc: Steven Rostedt <rostedt@goodmis.org> Cc: Keiichi KII <k-keiichi@bx.jp.nec.com> Cc: Peter Zijlstra <a.p.zijlstra@chello.nl> Cc: Paul Mackerras <paulus@samba.org> Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
* perf/scripts: Add perf-trace-python DocumentationTom Zanussi2010-02-24
| | | | | | | | | | | | | | Also small update to perf-trace-perl and perf-trace docs. Signed-off-by: Tom Zanussi <tzanussi@gmail.com> Cc: Ingo Molnar <mingo@elte.hu> Cc: Steven Rostedt <rostedt@goodmis.org> Cc: Keiichi KII <k-keiichi@bx.jp.nec.com> Cc: Peter Zijlstra <a.p.zijlstra@chello.nl> Cc: Paul Mackerras <paulus@samba.org> Cc: Arnaldo Carvalho de Melo <acme@redhat.com> LKML-Reference: <1264580883-15324-13-git-send-email-tzanussi@gmail.com> Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
* perf/scripts: Fix supported language listing optionTom Zanussi2010-02-23
| | | | | | | | | | | | | | | | | | | 'perf trace -s list' prints a list of the supported scripting languages. One problem with it is that it falls through and prints the trace as well. The use of 'list' for this also makes it easy to confuse with 'perf trace -l', used for listing available scripts. So change 'perf trace -s list' to 'perf trace -s lang' and fixes the fall-through problem. Signed-off-by: Tom Zanussi <tzanussi@gmail.com> Cc: Ingo Molnar <mingo@elte.hu> Cc: Steven Rostedt <rostedt@goodmis.org> Cc: Keiichi KII <k-keiichi@bx.jp.nec.com> Cc: Peter Zijlstra <a.p.zijlstra@chello.nl> Cc: Paul Mackerras <paulus@samba.org> Cc: Arnaldo Carvalho de Melo <acme@redhat.com> LKML-Reference: <1264580883-15324-2-git-send-email-tzanussi@gmail.com> Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
* perf top: Teach it to autolocate vmlinuxKirill Smelkov2010-02-04
| | | | | | | | | | | | | | | | | | | | | | | By relying on logic in dso__load_kernel_sym(), we can automatically load vmlinux. The only thing which needs to be adjusted, is how --sym-annotate option is handled - now we can't rely on vmlinux been loaded until full successful pass of dso__load_vmlinux(), but that's not the case if we'll do sym_filter_entry setup in symbol_filter(). So move this step right after event__process_sample() where we know the whole dso__load_kernel_sym() pass is done. By the way, though conceptually similar `perf top` still can't annotate userspace - see next patches with fixes. Signed-off-by: Kirill Smelkov <kirr@landau.phys.spbu.ru> Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com> Cc: Mike Galbraith <efault@gmx.de> LKML-Reference: <1265223128-11786-9-git-send-email-acme@infradead.org> Signed-off-by: Ingo Molnar <mingo@elte.hu>
* perf buildid-cache: Add new command to manage build-id cacheArnaldo Carvalho de Melo2010-01-21
| | | | | | | | | | | | | | | | | | | For now it just has operations to examine a given file, find its build-id and add or remove it to/from the cache. Useful, for instance, when adding binaries sent together with a perf.data file, so that we can add them to the cache and have the tools find it when resolving symbols. It'll also manage the size of the cache like 'ccache' does. Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com> Cc: Frédéric Weisbecker <fweisbec@gmail.com> Cc: Mike Galbraith <efault@gmx.de> Cc: Peter Zijlstra <a.p.zijlstra@chello.nl> Cc: Paul Mackerras <paulus@samba.org> LKML-Reference: <1264008525-29025-1-git-send-email-acme@infradead.org> Signed-off-by: Ingo Molnar <mingo@elte.hu>
* perf archive: Add documentationArnaldo Carvalho de Melo2010-01-20
| | | | | | | | | | | | | This also makes it appear on the 'perf --help' output, i.e. util/generate-cmdlist.sh now takes it into account. Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com> Cc: Frédéric Weisbecker <fweisbec@gmail.com> Cc: Mike Galbraith <efault@gmx.de> Cc: Peter Zijlstra <a.p.zijlstra@chello.nl> Cc: Paul Mackerras <paulus@samba.org> LKML-Reference: <1263837559-24168-1-git-send-email-acme@infradead.org> Signed-off-by: Ingo Molnar <mingo@elte.hu>
* perf: Fix few typos + cosmeticsKirill Smelkov2010-01-13
| | | | | | | Signed-off-by: Kirill Smelkov <kirr@landau.phys.spbu.ru> Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com> LKML-Reference: <1263396139-4798-2-git-send-email-acme@infradead.org> Signed-off-by: Ingo Molnar <mingo@elte.hu>
* perf probe: Support --line option to show probable source-code linesMasami Hiramatsu2010-01-13
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Add --line option to support showing probable source-code lines. perf probe --line SRC:LN[-LN|+NUM] or perf probe --line FUNC[:LN[-LN|+NUM]] This option shows source-code with line number if the line can be probed. Lines without line number (and blue color) means that the line can not be probed, because debuginfo doesn't have the information of those lines. The argument specifies the range of lines, "source.c:100-120" shows lines between 100th to l20th in source.c file. And "func:10+20" shows 20 lines from 10th line of func function. e.g. # ./perf probe --line kernel/sched.c:1080 <kernel/sched.c:1080> * * called with rq->lock held and irqs disabled */ static void hrtick_start(struct rq *rq, u64 delay) { struct hrtimer *timer = &rq->hrtick_timer; 1086 ktime_t time = ktime_add_ns(timer->base->get_time(), delay); hrtimer_set_expires(timer, time); 1090 if (rq == this_rq()) { 1091 hrtimer_restart(timer); 1092 } else if (!rq->hrtick_csd_pending) { 1093 __smp_call_function_single(cpu_of(rq), &rq->hrtick_csd, 1094 rq->hrtick_csd_pending = 1; If you specifying function name, this shows function-relative line number. # ./perf probe --line schedule <schedule:0> asmlinkage void __sched schedule(void) 1 { struct task_struct *prev, *next; unsigned long *switch_count; struct rq *rq; int cpu; need_resched: preempt_disable(); 9 cpu = smp_processor_id(); 10 rq = cpu_rq(cpu); 11 rcu_sched_qs(cpu); 12 prev = rq->curr; 13 switch_count = &prev->nivcsw; Signed-off-by: Masami Hiramatsu <mhiramat@redhat.com> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: Arnaldo Carvalho de Melo <acme@redhat.com> Cc: systemtap <systemtap@sources.redhat.com> Cc: DLE <dle-develop@lists.sourceforge.net> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: Paul Mackerras <paulus@samba.org> Cc: Arnaldo Carvalho de Melo <acme@redhat.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Mike Galbraith <efault@gmx.de> LKML-Reference: <20100106144534.27218.77939.stgit@dhcp-100-2-132.bos.redhat.com> Signed-off-by: Ingo Molnar <mingo@elte.hu>
* perf diff: Use perf_session__fprintf_hists just like 'perf record'Arnaldo Carvalho de Melo2009-12-16
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | That means that almost everything you can do with 'perf report' can be done with 'perf diff', for instance: $ perf record -f find / > /dev/null [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.062 MB perf.data (~2699 samples) ] $ perf record -f find / > /dev/null [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.062 MB perf.data (~2687 samples) ] perf diff | head -8 9.02% +1.00% find libc-2.10.1.so [.] _IO_vfprintf_internal 2.91% -1.00% find [kernel] [k] __kmalloc 2.85% -1.00% find [kernel] [k] ext4_htree_store_dirent 1.99% -1.00% find [kernel] [k] _atomic_dec_and_lock 2.44% find [kernel] [k] half_md4_transform $ So if you want to zoom into libc: $ perf diff --dsos libc-2.10.1.so | head -8 37.34% find [.] _IO_vfprintf_internal 10.34% find [.] __GI_memmove 8.25% +2.00% find [.] _int_malloc 5.07% -1.00% find [.] __GI_mempcpy 7.62% +2.00% find [.] _int_free $ And if there were multiple commands using libc, it is also possible to aggregate them all by using --sort symbol: $ perf diff --dsos libc-2.10.1.so --sort symbol | head -8 37.34% [.] _IO_vfprintf_internal 10.34% [.] __GI_memmove 8.25% +2.00% [.] _int_malloc 5.07% -1.00% [.] __GI_mempcpy 7.62% +2.00% [.] _int_free $ The displacement column now is off by default, to use it: perf diff -m --dsos libc-2.10.1.so --sort symbol | head -8 37.34% [.] _IO_vfprintf_internal 10.34% [.] __GI_memmove 8.25% +2.00% [.] _int_malloc 5.07% -1.00% +2 [.] __GI_mempcpy 7.62% +2.00% -1 [.] _int_free $ Using -t/--field-separator can be used for scripting: $ perf diff -t, -m --dsos libc-2.10.1.so --sort symbol | head -8 37.34, , ,[.] _IO_vfprintf_internal 10.34, , ,[.] __GI_memmove 8.25,+2.00%, ,[.] _int_malloc 5.07,-1.00%, +2,[.] __GI_mempcpy 7.62,+2.00%, -1,[.] _int_free 6.99,+1.00%, -1,[.] _IO_new_file_xsputn 1.89,-2.00%, +4,[.] __readdir64 $ Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com> Cc: Frédéric Weisbecker <fweisbec@gmail.com> Cc: Mike Galbraith <efault@gmx.de> Cc: Peter Zijlstra <a.p.zijlstra@chello.nl> Cc: Paul Mackerras <paulus@samba.org> LKML-Reference: <1260978567-550-1-git-send-email-acme@infradead.org> Signed-off-by: Ingo Molnar <mingo@elte.hu>