1 <!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.1.2//EN" "http://www.oasis-open.org/docbook/xml/4.1.2/docbookx.dtd">
3 <book id="oprofile-guide">
5 <title>OProfile manual</title>
9 <firstname>John</firstname>
10 <surname>Levon</surname>
12 <address><email>levon@movementarian.org</email></address>
18 <year>2000-2004</year>
19 <holder>Victoria University of Manchester, John Levon and others</holder>
25 <chapter id="introduction">
26 <title>Introduction</title>
29 This manual applies to OProfile version <oprofileversion />.
30 OProfile is a set of performance monitoring tools for Linux 2.6 and higher systems, available on a number of architectures.
31 OProfile provides the following features:
33 <listitem>Profiler</listitem>
34 <listitem>Post-processing tools for analyzing profile data</listitem>
35 <listitem>Event counter</listitem>
39 OProfile is capable of monitoring native hardware events occurring in all parts of a running system, from the kernel
40 (including modules and interrupt handlers) to shared libraries
41 to binaries. OProfile can collect event information for the whole system in the background with very little overhead. These
42 features make it ideal for monitoring entire systems to determine bottle necks in real-world systems.
46 Many CPUs provide "performance counters", hardware registers that can count "events"; for example,
47 cache misses, or CPU cycles. OProfile can collect profiles of code based on the number of these occurring events:
48 repeatedly, every time a certain (configurable) number of events has occurred, the PC value is recorded.
49 This information is aggregated into profiles for each binary image. Alternatively, OProfile's event counting
50 tool can collect simple raw event counts.</para>
51 <sect1 id="legacy_mode">
52 <title>OProfile legacy profiling mode</title>
53 Prior to release 1.0, OProfile included a profiling tool consisting of the <command>opcontrol</command> shell script, the <command>oprofiled</command> daemon,
54 and the attendant oprofile kernel driver. This "legacy profiler" was deprecated in release 0.9.8 with the introduction of
55 the <command>operf</command> profiling tool (see <xref linkend="perf_events"/>). Some older architectures/platforms
56 do not support the use of <command>operf</command>. For those cases, oprofile users should install release 0.9.9, which is the
57 last release to include the legacy profiler.
59 <sect1 id="perf_events">
60 <title>OProfile perf_events profiling mode</title>
62 OProfile has the ability to profile a single process or every currently running process (i.e., system-wide)
63 via the <command>operf</command> program. <command>operf</command> interfaces with the
64 kernel to collect samples via the Linux Kernel Performance Events Subsystem (hereafter
65 referred to as "perf_events"). OProfile can co-exist with other tools on your system that
66 may also be using the perf_events kernel subsystem.
69 Using <command>operf</command> to profile a single
70 process can be done as a normal user; however, root authority <emphasis>is</emphasis> required to run
71 <command>operf</command> in system-wide profiling mode.
74 Some older processor models are not supported by the underlying perf_events kernel and, thus, are not supported by <command>operf</command>.
75 If you receive the message
76 <screen> Your kernel's Performance Events Subsystem does not support your processor type</screen>
77 when attempting to use <command>operf</command>, install OProfile 0.9.9 and try profiling with <command>opcontrol</command>
78 to see if your processor type may be supported by OProfile's legacy mode.
83 <sect1 id="event_counting">
84 <title>OProfile event counting mode</title>
85 OProfile provides the <command>ocount</command> tool for
86 collecting raw event counts on a per-application, per-process, per-cpu, or system-wide basis. Unlike the
87 profiling tools, post-processing of the data collected is not necessary -- the data is displayed in the
88 output of <command>ocount</command>. A common use case for event counting tools is when performance analysts
89 want to determine the CPI (cycles per instruction) for an application. High CPI implies possible stalls,
90 and many architectures provide events that give detailed information about the different types of stalls.
91 The events provided are architecture-specific, so we refer the reader to the hardware manuals available for
92 the processor type being used.
96 <sect1 id="applications">
97 <title>Applications of OProfile</title>
99 OProfile is useful in a number of situations. You might want to use OProfile when you :
102 <listitem><para>need low overhead</para></listitem>
103 <listitem><para>cannot use highly intrusive profiling methods</para></listitem>
104 <listitem><para>need to profile interrupt handlers</para></listitem>
105 <listitem><para>need to profile an application and its shared libraries</para></listitem>
106 <listitem><para>need to profile dynamically compiled code of supported virtual machines (see <xref linkend="jitsupport"/>)</para></listitem>
107 <listitem><para>need to capture the performance behaviour of entire system</para></listitem>
108 <listitem><para>want to examine hardware effects such as cache misses</para></listitem>
109 <listitem><para>want detailed source annotation</para></listitem>
110 <listitem><para>want instruction-level profiles</para></listitem>
111 <listitem><para>want call-graph profiles</para></listitem>
114 OProfile is not a panacea. OProfile might not be a complete solution when you :
117 <listitem><para>require call graph profiles on platforms other than x86, ARM, and PowerPC</para></listitem>
118 <listitem><para>require 100% instruction-accurate profiles</para></listitem>
119 <listitem><para>need function call counts or an interstitial profiling API</para></listitem>
120 <listitem><para>cannot tolerate any disturbance to the system whatsoever</para></listitem>
121 <listitem><para>need to profile interpreted or dynamically compiled code of non-supported virtual machines</para></listitem>
123 <sect2 id="jitsupport">
124 <title>Support for dynamically compiled (JIT) code</title>
126 OProfile provides a framework to support JITed code ("just-in-time (JIT) compiled code").
127 A development library is provided to allow developers
128 to add support for any VM (virtual machine) that produces dynamically compiled code (see the <emphasis>OProfile JIT agent
129 developer guide</emphasis>).
130 In addition, built-in support is included for the following:</para>
131 <itemizedlist><listitem>JVMTI agent library for Java (1.5 and higher)</listitem>
132 <listitem>JVMPI agent library for Java (1.5 and lower)</listitem>
134 These libraries make it possible for OProfile to attribute profile samples
135 to Java methods. Without a VM-specific agent library, OProfile will typically report
136 samples from JITed code similar to the following example:
137 <screen> anon: <tgid><address range></screen>
138 For information on how to use OProfile's JIT support, see <xref linkend="setup-jit"/>.
141 <sect2 id="guestsupport">
142 <title>No support for virtual machine guests</title>
144 OProfile currently does not support event-based profiling (i.e, using hardware events like cache misses,
145 branch mispredicts) on virtual machine guests running under systems such as VMware.
146 (Note: KVM guests <emphasis>are</emphasis> supported.) The list of
147 supported events displayed by ophelp is based on CPU type and does
148 not take into account whether the running system is a guest system or real system. To use
149 OProfile on such guest systems, you must use the legacy profiler's timer mode (see <xref linkend="timer" />).
156 <sect1 id="requirements">
157 <title>System requirements</title>
161 <term>Linux kernel</term>
163 Release 2.6.31 or higher
167 <term>Supported architectures</term>
169 AMD, ARM, Intel, PowerPC, Tile, MIPS
173 <term>Required libraries</term>
175 These libraries are required : <filename>popt</filename>, <filename>bfd</filename>,
176 <filename>liberty</filename> (debian users: libiberty is provided in binutils-dev package), <filename>dl</filename>,
177 plus the standard C++ libraries.
181 <term>Required kernel headers</term>
183 Either the kernel-headers package must be installed or use the <code>--with-kernel</code>
188 <term>Required user account</term>
190 For secure processing of sample data from JIT virtual machines (e.g., Java),
191 the special user account "oprofile" must exist on the system. The 'configure'
192 and 'make install' operations will print warning messages if this
193 account is not found. If you intend to profile JITed code, you must create
194 a group account named 'oprofile' and then create the 'oprofile' user account,
195 setting the default group to 'oprofile'. A runtime error message is printed to
196 the oprofile log when processing JIT samples if this special user
197 account cannot be found.
201 <term><acronym>ELF</acronym></term>
203 Probably not too strenuous a requirement, but older <acronym>A.OUT</acronym> binaries/libraries are not supported.
207 <term>K&R coding style</term>
209 OK, so it's not really a requirement, but I wish it was...
217 <sect1 id="resources">
218 <title>Internet resources</title>
222 <term>Web page</term>
224 There is a web page (which you may be reading now) at
225 <ulink url="http://oprofile.sf.net/">http://oprofile.sf.net/</ulink>.
229 <term>Download</term>
231 You can download a source tarball or check out code from
232 the code repository at the sourceforge page,
233 <ulink url="http://sf.net/projects/oprofile/">http://sf.net/projects/oprofile/</ulink>.
237 <term>Mailing list</term>
239 There is a low-traffic OProfile-specific mailing list, details at
240 <ulink url="http://sf.net/mail/?group_id=16191">http://sf.net/mail/?group_id=16191</ulink>.
244 <term>Bug tracker</term>
246 There is a bug tracker for OProfile at SourceForge,
247 <ulink url="http://sourceforge.net/p/oprofile/bugs/">http://sourceforge.net/p/oprofile/bugs/</ulink>.
251 <term>IRC channel</term>
253 Several OProfile developers and users sometimes hang out on channel <command>#oprofile</command>
254 on the <ulink url="http://oftc.net">OFTC</ulink> network.
262 <title>Installation</title>
265 First you need to build OProfile and install it. <command>./configure</command>, <command>make</command>, <command>make install</command>
266 is often all you need, but note these arguments to <command>./configure</command> :
270 <term><option>--with-java</option></term>
273 Use this option if you need to profile Java applications. Also, see
274 <xref linkend="requirements"/>, "Required user account". This option
275 is used to specify the location of the Java Development Kit (JDK)
276 source tree you wish to use. This is necessary to get the interface description
277 of the JVMPI (or JVMTI) interface to compile the JIT support code successfully.
281 The Java Runtime Environment (JRE) does not include the development
282 files that are required to compile the JIT support code, so the full
283 JDK must be installed in order to use this option.
287 By default, the Oprofile JIT support libraries will be installed in
288 <filename><oprof_install_dir>/lib/oprofile</filename>. To build
289 and install OProfile and the JIT support libraries as 64-bit, you can
290 do something like the following:
292 # CFLAGS="-m64" CXXFLAGS="-m64" ./configure \
293 --with-java={my_jdk_installdir} \
294 --libdir=/usr/local/lib64
299 If you encounter errors building 64-bit, you should
300 install libtool 1.5.26 or later since that release of
301 libtool fixes known problems for certain platforms.
302 If you install libtool into a non-standard location,
303 you'll need to edit the invocation of 'aclocal' in
304 OProfile's autogen.sh as follows (assume an install
305 location of /usr/local):
308 <code>aclocal -I m4 -I /usr/local/share/aclocal</code>
313 <varlistentry id="disable-werror">
314 <term><option>--disable-werror</option></term>
316 Development versions of OProfile build by
317 default with <option>-Werror</option>. This option turns
318 <option>-Werror</option> off.
321 <varlistentry id="disable-optimization">
322 <term><option>--disable-optimization</option></term>
324 Disable the <option>-O2</option> compiler flag
325 (useful if you discover an OProfile bug and want to give a useful
329 <varlistentry id="with-kernel">
330 <term><option>--with-kernel</option></term>
332 This option is used to specify the location of the kernel headers <filename>include</filename> directory
333 needed to build the perf_events-enabled <command>operf</command> program. By default, the OProfile
334 build system expects to find this directory under <filename>/usr</filename>. Use this option if your
335 kernel headers are in a non-standard location or if building in a cross-compile enviroment or in a
336 situation where the host system does not support perf_events but you wish to build binaries for a
337 target system that does support perf_events.
342 It is recommended that if you have a
343 uniprocessor machine, you enable the local APIC / IO_APIC support for
344 your kernel (this is automatically enabled for SMP kernels). With many BIOS (kernel >= 2.6.9 and UP kernel)
345 it's not sufficient to enable the local APIC -- you must also turn it on explicitly at boot
346 time by providing the "lapic" option to the kernel.
347 If you use the NMI watchdog, be aware that the watchdog is disabled when profiling starts
348 and not re-enabled until the profiling is stopped.
353 <sect1 id="uninstall">
354 <title>Uninstalling OProfile</title>
356 You must have the source tree available to uninstall OProfile; a <command>make uninstall</command> will
357 remove all installed files except your configuration file in the directory <filename>~/.oprofile</filename>.
363 <chapter id="overview">
364 <title>Overview</title>
365 <sect1 id="getting-started-with-operf">
366 <title>Getting started with OProfile using <command>operf</command></title>
368 Profiling with <command>operf</command> allows you to precisely target your profiling (i.e., single process
369 or system-wide). With <command>operf</command>, there is no initial setup needed -- simply invoke <command>operf</command> with
370 the options you need; then run the OProfile post-processing tool(s). The <command>operf</command> syntax
373 <screen>operf [ options ] [ --system-wide | --pid=<PID> | [ command [ args ] ] ]</screen>
375 A typical usage might look like this:
377 <screen>operf ./my_test_program my_arg</screen>
379 When <filename>./my_test_program</filename> completes (or when you press Ctrl-C), profiling
380 stops and you're ready to use <command>opreport</command> or other OProfile post-processing tools.
381 By default, <command>operf</command> stores the sample data in <filename><cur_dir>/oprofile_data/samples/current</filename>,
382 and <command>opreport</command> and other post-processing tools will look in that location first for profile data,
383 unless you pass the <code>--session-dir</code> option.
388 <sect1 id="getting-started-with-ocount">
389 <title>Getting started with OProfile using <command>ocount</command></title>
391 <command>ocount</command> is an OProfile tool that can be used to count native hardware events occurring in either
392 a specific application, a set of processes or threads, a set of active system processors, or the
393 entire system. The data collected during a counting session is displayed to stdout by default, but may
394 also be saved to a file. The <command>ocount</command> syntax is as follows:
396 <screen>ocount [ options ] [ --system-wide | --process-list <pids> | --thread-list <tids> | --cpu-list <cpus> [ command [ args ] ] ]
399 A typical usage might look like this:
401 <screen>ocount --events=CPU_CLK_UNHALTED,INST_RETIRED /home/user1/my_test_program my_arg</screen>
403 When <filename>my_test_program</filename> completes (or when you press Ctrl-C), counting
404 stops and the results are displayed to the screen (as shown below).
407 Events were actively counted for 2.8 seconds.
408 Event counts (actual) for /home/user1/my_test_program:
409 Event Count % time counted
410 CPU_CLK_UNHALTED 9,408,018,070 100.00
411 INST_RETIRED 16,719,918,108 100.00
417 <sect1 id="eventspec">
418 <title>Specifying performance counter events</title>
420 Whether profiling with <command>operf</command> or doing simple event counting with <command>ocount</command>,
421 you can collect information about one more native hardware events using the <code>--events</code>
422 option -- a comma-separated list of event specfications. The event specification is the means to provide details
423 of how each hardware performance counter should be set up.
424 For profiling, the event specification is a colon-separated string of the form
425 <option><emphasis>name</emphasis>:<emphasis>count</emphasis>:<emphasis>unitmask</emphasis>:<emphasis>kernel</emphasis>:<emphasis>user</emphasis></option>
426 as described in the table below. For <command>ocount</command>, specification is of the form
427 <option><emphasis>name</emphasis>:<emphasis>unitmask</emphasis>:<emphasis>kernel</emphasis>:<emphasis>user</emphasis></option>.
428 Note the presence of the <emphasis>count</emphasis> field for profiling. The <emphasis>count</emphasis> field tells the profiler
429 how many events should occur between a profile snapshot (usually referred to as a "sample"). Since
430 <command>ocount</command> does not do sampling, the <emphasis>count</emphasis> field is not needed.
433 If no event specs are passed to <command>operf</command> or <command>ocount</command>,
434 the default event will be used.
437 <note>The perf_events kernel subsystem allocates hardware counters as necessary, but some processor
438 types have restrictions as to what hardware events may be counted simultaneously.
439 The kernel employs a multiplexing technique when such
440 hardware restrictions are encountered, such that events are monitored on a rotating basis.
443 <informaltable frame="all">
446 <row><entry><option>name</option></entry><entry>The symbolic event name, e.g. <constant>CPU_CLK_UNHALTED</constant></entry></row>
447 <row><entry><option>count</option></entry><entry>The counter reset value, e.g. 100000; use only for profiling</entry></row>
448 <row><entry><option>unitmask</option></entry><entry>The unit mask, as given in the events list: e.g. 0x0f; or a symbolic name
449 if a <constant>name=<um_name></constant> field is present</entry></row>
450 <row><entry><option>kernel</option></entry><entry>Enable profiling of kernel code</entry></row>
451 <row><entry><option>user</option></entry><entry>Enable profiling of userspace code</entry></row>
456 The last three values are optional; if you omit them (e.g. <option>operf --events=DATA_MEM_REFS:30000</option>),
457 they will be set to the default values (i.e., the default unit mask value for the given event, and profiling (or counting)
458 both kernel and userspace code will be enabled). Note that on some architectures, some events may
459 require a unit mask be specified.
462 You can specify unit mask values using either a numerical value (hex values
463 <emphasis>must</emphasis> begin with "0x") or a symbolic name (if the <constant>name=<um_name></constant>
464 field is shown in the <command>ophelp</command> output). For some named unit masks, the hex value is not unique; thus, OProfile
465 tools enforce specifying such unit masks value by name.
468 The table below lists the default profiling event for various processor types. The same events
469 can be used for <command>ocount</command>, minus the <emphasis>count</emphasis> field.
471 <informaltable frame="all">
474 <row><entry>Processor</entry><entry>cpu_type</entry><entry>Default event</entry></row>
475 <row><entry>Alpha EV67</entry><entry>alpha/ev67</entry><entry>CYCLES:100000:0:1:1</entry></row>
476 <row><entry>ARM/XScale PMU1</entry><entry>arm/xscale1</entry><entry>CPU_CYCLES:100000:0:1:1</entry></row>
477 <row><entry>ARM/XScale PMU2</entry><entry>arm/xscale2</entry><entry>CPU_CYCLES:100000:0:1:1</entry></row>
478 <row><entry>ARM/MPCore</entry><entry>arm/mpcore</entry><entry>CPU_CYCLES:100000:0:1:1</entry></row>
479 <row><entry>Athlon</entry><entry>i386/athlon</entry><entry>CPU_CLK_UNHALTED:100000:0:1:1</entry></row>
480 <row><entry>Pentium Pro</entry><entry>i386/ppro</entry><entry>CPU_CLK_UNHALTED:100000:0:1:1</entry></row>
481 <row><entry>Pentium II</entry><entry>i386/pii</entry><entry>CPU_CLK_UNHALTED:100000:0:1:1</entry></row>
482 <row><entry>Pentium III</entry><entry>i386/piii</entry><entry>CPU_CLK_UNHALTED:100000:0:1:1</entry></row>
483 <row><entry>Pentium M (P6 core)</entry><entry>i386/p6_mobile</entry><entry>CPU_CLK_UNHALTED:100000:0:1:1</entry></row>
484 <row><entry>Pentium 4 (non-HT)</entry><entry>i386/p4</entry><entry>GLOBAL_POWER_EVENTS:100000:1:1:1</entry></row>
485 <row><entry>Pentium 4 (HT)</entry><entry>i386/p4-ht</entry><entry>GLOBAL_POWER_EVENTS:100000:1:1:1</entry></row>
486 <row><entry>Hammer</entry><entry>x86-64/hammer</entry><entry>CPU_CLK_UNHALTED:100000:0:1:1</entry></row>
487 <row><entry>Family10h</entry><entry>x86-64/family10</entry><entry>CPU_CLK_UNHALTED:100000:0:1:1</entry></row>
488 <row><entry>Family11h</entry><entry>x86-64/family11h</entry><entry>CPU_CLK_UNHALTED:100000:0:1:1</entry></row>
489 <row><entry>IBM pseries</entry><entry>ppc64/power{ 4|5|6|7|8|970 }</entry><entry>CYCLES:100000:0:1:1</entry></row>
490 <row><entry>IBM s390</entry><entry>s390/{ z10|z196|zEC12 }</entry><entry>HWSAMPLING:4127518:0:1:1</entry></row>
497 <sect1 id="tools-overview">
498 <title>Tools summary</title>
500 This section gives a brief description of the available OProfile utilities and their purpose.
504 <term><filename>ophelp</filename></term>
506 This utility lists the available events and short descriptions.
511 <term><filename>operf</filename></term>
513 This is the program for collecting profile data, discussed in <xref linkend="controlling-operf" />.
518 <term><filename>ocount</filename></term>
520 This tool is used for simple event counting, as described in in <xref linkend="controlling-ocount" />.
525 <term><filename>agent libraries</filename></term>
527 Used by virtual machines (like the Java VM) to record information about JITed code being profiled. See <xref linkend="setup-jit" />.
532 <term><filename>opreport</filename></term>
534 This is the main tool for retrieving useful profile data, described in
535 <xref linkend="opreport" />.
540 <term><filename>opannotate</filename></term>
542 This utility can be used to produce annotated source, assembly or mixed source/assembly.
543 Source level annotation is available only if the application was compiled with
544 debugging symbols. See <xref linkend="opannotate" />.
549 <term><filename>opgprof</filename></term>
551 This utility can output gprof-style data files for a binary, for use with
552 <command>gprof -p</command>. See <xref linkend="opgprof" />.
557 <term><filename>oparchive</filename></term>
559 This utility can be used to collect executables, debuginfo,
560 and sample files and copy the files into an archive.
561 The archive is self-contained and can be moved to another
562 machine for further analysis.
563 See <xref linkend="oparchive" />.
568 <term><filename>opimport</filename></term>
570 This utility converts sample database files from a foreign binary format (abi) to
571 the native format. This is useful only when moving sample files between hosts
572 for analysis on platforms other than the one used for collection.
573 See <xref linkend="opimport" />.
582 <chapter id="controlling-profiler">
583 <title>Controlling the profiler</title>
585 <sect1 id="controlling-operf">
586 <title>Using <command>operf</command></title>
588 This section describes in detail how <command>operf</command> is used to
589 control profiling. Unless otherwise directed, <command>operf</command> will profile using
590 the default event for your system. For most systems, the default event is some
591 cycles-based event, assuming your processor type supports hardware performance
592 counters. If your hardware <emphasis>does</emphasis> support performance counters, you can specify
593 something other than the default hardware event on which to profile. The performance
594 monitor counters can be programmed to count various hardware events,
595 such as cache misses or MMX operations. The event
596 chosen for each counter is reflected in the profile data collected
597 by OProfile: functions and binaries at the top of the profiles reflect
598 that most of the chosen events happened within that code.
601 Additionally, each counter is programmed with a "count" value, which corresponds to how
602 detailed the profile is. The lower the value, the more frequently profile
603 samples are taken. You can choose to sample only kernel code, user-space code,
604 or both (both is the default). Finally, some events have a "unit mask"
605 -- this is a value that further restricts the type of event being counted.
606 You can see the event types and unit masks for your CPU using <command>ophelp</command>.
607 More information on event specification can be found at <xref linkend="eventspec"/>.
610 The <command>operf</command> command syntax is:
612 <screen>operf [ options ] [ --system-wide | --pid=<PID> | [ command [ args ] ] ]</screen>
614 When profiling an application using either the <code>command</code> or <code>--pid</code> option of
615 <command>operf</command>, forks and execs of the profiled process will also be profiled. The samples
616 from an exec'ed process will be attributed to the executable binary run by that process. See
617 <xref linkend="interpreting_operf_results"/>
620 Following is a description of the <command>operf</command> options.
624 <term><option>command [args]</option></term>
626 The command or application to be profiled. The <emphasis>[args]</emphasis> are the input arguments
627 that the command or application requires. Either <code>command</code>, <code>--pid</code> or
628 <code>--system-wide</code> is required, but cannot be used simultaneously.
632 <term><option>--pid / -p [PID]</option></term>
634 This option enables <command>operf</command> to profile a running application. <code>PID</code>
635 should be the process ID of the process you wish to profile. When
636 finished profiling (e.g., when the profiled process ends), press
637 Ctrl-c to stop <command>operf</command>.
641 <term><option>--system-wide / -s</option></term>
643 This option is for performing a system-wide profile. You must
644 have root authority to run <command>operf</command> in this mode.
645 When finished profiling, Ctrl-C to stop <command>operf</command>. If you run
646 <code>operf --system-wide</code> as a background job (i.e., with the &), you
647 <emphasis>must</emphasis> stop it in a controlled manner in order to process
648 the profile data it has collected. Use <code>kill -SIGINT <operf-PID></code>
649 for this purpose. It is recommended that when running <command>operf</command>
650 with this option, your current working directory should be <filename>/root</filename> or a subdirectory
651 of <filename>/root</filename> to avoid storing sample data files in locations accessible by regular users.
655 <term><option>--vmlinux / k [vmlinux_path]</option></term>
657 A vmlinux file that matches the running kernel that has symbol and/or debuginfo.
658 Kernel samples will be attributed to this binary, allowing post-processing tools
659 (like <command>opreport</command>) to attribute samples to the appropriate kernel symbols.
660 If this option is not specified, the file /proc/kallsyms is used to obtain
661 kernel symbol addresses correponding to sample addresses. However, the setting of
662 /proc/sys/kernel/kptr_restrict may restrict a non-root user's access to
663 /proc/kallsyms, in which case,
664 all kernel samples are attributed to a pseudo binary named "no-vmlinux".
668 <term><option>--callgraph / -g</option></term>
670 This option enables the callgraph to be saved during profiling. NOTE: The
671 full callchain is recorded, so there is no depth limit.
675 <term><option>--append / -a</option></term>
677 By default, <command>operf</command> moves old profile data from
678 <filename><session_dir>/samples/current</filename> to
679 <filename><session_dir>/samples/previous</filename>.
680 If a 'previous' profile already existed, it will be replaced. If the
681 <code>--append</code> option is passed, old profile data in 'current' is left in place and
682 new profile data will be added to it, and the 'previous' profile (if one existed)
683 will remain untouched. To access the 'previous' profile, simply add a session
684 specification to the normal invocation of oprofile post-processing tools; for example:
687 <screen>opreport session:previous</screen>
691 <term><option>--events / -e [event1[,event2[,...]]]</option></term>
693 This option is for passing a comma-separated list of event specifications
694 for profiling. Each event spec is of the form:
696 <screen>name:count[:unitmask[:kernel[:user]]]</screen>
698 When no event specification is given, the default event for the running
699 processor type will be used for profiling. Use <command>ophelp</command>
700 to list the available events for your processor type.
704 <term><option>--separate-thread / -t</option></term>
706 This option categorizes samples by thread group ID (tgid) and thread ID (tid).
707 The <code>--separate-thread</code> option is useful for seeing per-thread samples in
708 multi-threaded applications. When used in conjuction with the <code>--system-wide</code>
709 option, <code>--separate-thread</code> is also useful for seeing per-process
710 (i.e., per-thread group) samples for the case where multiple processes are
711 executing the same program during a profiling run.
715 <term><option>--separate-cpu / -c</option></term>
717 This option categorizes samples by cpu.
721 <term><option>--session-dir / -d [path]</option></term>
723 This option specifies the session directory to hold the sample data. If not specified,
724 the data is saved in the <filename>oprofile_data</filename> directory on the current path.
728 <term><option>---lazy-conversion / -l</option></term>
730 Use this option to reduce the overhead of <command>operf</command> during profiling.
731 Normally, profile data received from the kernel is converted to OProfile format
732 during profiling time. This is typically not an issue when profiling a single
733 application. But when using the <code>--system-wide</code> option, this on-the-fly
734 conversion process can cause noticeable overhead, particularly on busy
735 multi-processor systems. The <code>--lazy-conversion</code> option directs
736 <command>operf</command> to wait until profiling is completed to do the conversion
741 <term><option>--verbose / -V [level]</option></term>
743 A comma-separated list of debugging control values used to increase the verbosity of the
744 output. Valid values are: debug, record, convert, misc, sfile, arcs, and the special value, 'all'.
748 <term><option>--version -v </option></term>
750 Show <command>operf</command> version.
754 <term><option>--help / -h</option></term>
763 <sect1 id="setup-jit">
764 <title>Setting up the JIT profiling feature</title>
766 To gather information about JITed code from a virtual machine,
767 it needs to be instrumented with an agent library. We use the
768 agent libraries for Java in the following example. To use the
769 Java profiling feature, you must build OProfile with the "--with-java" option
770 (<xref linkend="install" />).
774 <sect2 id="setup-jit-jvm">
775 <title>JVM instrumentation</title>
777 Add this to the startup parameters of the JVM (for JVMTI):
779 <screen><option>-agentpath:<libdir>/libjvmti_oprofile.so[=<options>]</option> </screen>
781 <screen><option>-agentlib:jvmti_oprofile[=<options>]</option> </screen>
784 The JVMPI agent implementation is enabled with the command line option
785 <screen><option>-Xrunjvmpi_oprofile[:<options>]</option> </screen>
788 Currently, there is just one option available -- <option>debug</option>. For JVMPI,
789 the convention for specifying an option is <option>option_name=[yes|no]</option>.
790 For JVMTI, the option specification is simply the option name, implying
791 "yes"; no option specified implies "no".
794 The agent library (installed in <filename><oprof_install_dir>/lib/oprofile</filename>)
795 needs to be in the library search path (e.g. add the library directory
796 to <constant>LD_LIBRARY_PATH</constant>). If the command line of
797 the JVM is not accessible, it may be buried within shell scripts or a
798 launcher program. It may also be possible to set an environment variable to add
800 For Sun JVMs this is <constant>JAVA_TOOL_OPTIONS</constant>. Please check
801 your JVM documentation for
802 further information on the agent startup options.
809 <sect1 id="detailed-parameters">
810 <title>Configuration details</title>
812 <sect2 id="hardware-counters">
813 <title>Hardware performance counters</title>
814 <para>Most processor models include performance monitor units that can be configured to monitor (count)
815 various types of hardware events. This section is where you can find architecture-specific information
816 to help you use these events for profiling. You do not really need to read this section unless you are interested in using
817 events other than the default event chosen by OProfile.
821 Your CPU type may not include the requisite support for hardware performance counters, in which case
822 you must use OProfile in timer mode (see <xref linkend="timer" />), which is only available in
823 OProfile releases prior to 1.0.
827 The Intel hardware performance counters are detailed in the Intel IA-32 Architecture Manual, Volume 3, available
828 from <ulink url="http://developer.intel.com/">http://developer.intel.com/</ulink>.
829 The AMD Athlon/Opteron/Phenom/Turion implementation is detailed in <ulink
830 url="http://www.amd.com/us-en/assets/content_type/white_papers_and_tech_docs/22007.pdf">
831 http://www.amd.com/us-en/assets/content_type/white_papers_and_tech_docs/22007.pdf</ulink>.
832 For IBM PowerPC processors, documentation is available at <ulink url="https://www.power.org/">
833 https://www.power.org/</ulink>. For example, <ulink url="https://www.power.org/events/Power7">
834 https://www.power.org/events/Power7</ulink> contains specific information on the performance
835 monitor unit for the IBM POWER7.
838 A physical performance monitor counter (PMC) is configured by a profiling tool to count a particular
839 type of event. When the counter overflows, an interrupt is delivered to the processor.
840 This is the basic mechanism on which OProfile is based. The delivery mode is <acronym>NMI</acronym>,
841 so blocking interrupts in the kernel does not prevent profiling. When the interrupt handler is called,
842 the current <acronym>PC</acronym> (program counter) value and the current task are recorded into the profiling structure.
843 This allows the overflow event to be attributed to a specific assembly instruction in a specific binary image.
844 OProfile receives this data (commonly referred to as a "sample") from the kernel and writes it to the sample files.
847 If we use an event such as <constant>CPU_CLK_UNHALTED</constant> or <constant>INST_RETIRED</constant>
848 (<constant>GLOBAL_POWER_EVENTS</constant> or <constant>INSTR_RETIRED</constant>, respectively, on the Pentium 4), we can
849 use the overflow counts (samples) as an estimate of actual time spent in each part of code. Alternatively we can profile interesting
850 data such as the cache behaviour of routines with the other available counters.
853 However there are several caveats. First, there are those issues listed in the Intel manual. There is a delay
854 between the counter overflow and the interrupt delivery that can skew results on a small scale - this means
855 you cannot rely on the profiles at the instruction level as being perfectly accurate.
856 For example, if you are profiling an application with an event that counts L1 cache misses, a sample attributed
857 to a particular instruction in the application doesn't necessarily mean that exact instruction is responsible
858 for that event; instead, it means the sample was taken in the dynamic vicinity of that instruction,
859 usually with a margin of error of a few instructions. Further details on this problem can be found in
860 <xref linkend="interpreting" /> and also in the Digital paper "ProfileMe: A Hardware Performance Counter".
863 Each counter has several configuration parameters besides the type of event to count.
864 First, there is the unit mask, which is used to further qualify exactly what to count.
865 Second, there is the <constant>count</constant> field, discussed below. Third, there are parameters
866 to specify whether to increment counts
867 whilst in kernel or user space. You can configure these separately for each counter.
870 When the profiler is initially setup, a performance monitor counter is chosen for counting the
871 event, and it is initialized using the <constant>count</constant> value.
872 Once profiling begins, the counter increments with each event detected, and the counter
873 <emphasis>overflows</emphasis> when the <constant>count</constant> value is reached.
874 As described above, the counter overflow generates an interrupt, and the sample is recorded.
875 After each overflow event, the counter is re-initialized using the <constant>count</constant> value,
876 and counting begins anew for the next sample. Higher values for <constant>count</constant>
877 result in samples being taken less frequently, and therefore less-detailed (and, potentially,
878 less accurate) profiling. Lower values mean more detail, but higher overhead.
879 Picking a good value for this parameter is, unfortunately, somewhat of a black art. It is
880 of course dependent on the event you have chosen.
881 Specifying too large a value will mean not enough interrupts are generated
882 to give a realistic profile (though this problem can be ameliorated by profiling for
883 longer time periods. Specifying too small a value can lead to higher performance overhead.
889 <title>OProfile timer interrupt mode</title>
891 Some CPU types do not provide the needed hardware support for hardware performance counters.
892 Additionally, some older architectures are not supported by the perf_events kernel subsystem.
893 On such machines, the <command>operf</command> and <command>ocount</command> commands will exit with a message indicating the
894 processor type is not supported. However, you can install OProfile 0.9.9 and use the legacy
895 opcontrol-based profiler, which will fall back to using timer interrupts for profiling.
896 Note that in timer mode, OProfile is not able to profile code that has interrupts disabled.
897 <note>Timer mode is only available using the legacy <command>opcontrol</command> command,
898 available in releases prior to 1.0.</note>
902 <sect2 id="special-notes">
903 <title>Architecture-specific configuration notes</title>
905 <title>Pentium 4 support</title>
907 The Pentium 4 / Xeon performance counters are organized around 3 types of model specific registers (MSRs): 45 event
908 selection control registers (ESCRs), 18 counter configuration control registers (CCCRs) and 18 counters. ESCRs describe a
909 particular set of events which are to be recorded, and CCCRs bind ESCRs to counters and configure their
910 operation. Unfortunately the relationship between these registers is quite complex; they cannot all be used with one
911 another at any time. There is, however, a subset of 8 counters, 8 ESCRs, and 8 CCCRs which can be used independently of
912 one another, so OProfile only accesses those registers, treating them as a bank of 8 "normal" counters, similar
913 to those in the P6 or Athlon/Opteron/Phenom/Turion families of CPU.
916 There is currently no support for Precision Event-Based Sampling (PEBS), nor any advanced uses of the Debug Store
917 (DS). Current support is limited to the conservative extension of OProfile's existing interrupt-based model described
923 <title>PowerPC64 support</title>
925 The performance monitoring unit (PMU) for the IBM PowerPC 64-bit processors
926 consists of between 4 and 8 counters (depending on the model). Advanced features
927 such as instruction matching and thresholding are not supported by OProfile.
939 <chapter id="results">
940 <title>Obtaining profiling results</title>
942 After collecting profile data, the raw data must undergo special processing in order for you to
943 perform your analysis. The analysis tools that perform this special processing are
944 <command>opreport</command>, <command>opannotate</command>, and <command>opgprof</command>.
945 Additionally, the <command>oparchive</command> is used to gather together profile
946 data, sampled binary files, etc. for the purpose of off-line analysis. While
947 not really an analysis tool, <command>oparchive</command> is put in that category
948 for convenience since it takes many of the same options as the other analysis tools.
951 <sect1 id="profile-spec">
952 <title>Profile specifications</title>
955 All of the analysis tools take a <emphasis>profile specification</emphasis>
956 as an input argument.
957 This is a set of definitions that describes the specific profile data that should be
958 examined. The simplest profile specification is empty: this will match all
959 the available profile files for the current session.
962 Specification parameters are of the form <option>name:value[,value]</option>.
963 For example, if I wanted to get a combined symbol summary for
964 <filename>/bin/myprog</filename> and <filename>/bin/myprog2</filename>,
965 I could do <command>opreport -l image:/bin/myprog,/bin/myprog2</command>.
966 As a special case, you don't actually need to specify the <option>image:</option>
967 part of the specification. Anything left on the command line after all other
968 <command>opreport</command> options have been processed is assumed to be an
969 <option>image:</option> name. Similarly, if no <option>session:</option>
970 is specified, then <option>session:current</option> is assumed ("current"
971 is a special name of the current (i.e., most recent) profiling session).
974 In addition to the comma-separated list shown above, some of the
975 specification parameters can take <command>glob</command>-style
976 values. For example, if I want to see image summaries for all
977 binaries profiled in <filename>/usr/bin/</filename>, I could do
978 <command>opreport image:/usr/bin/\*</command>. Note the necessity
979 to escape the special character from the shell.
982 For <command>opreport</command>, profile specifications can be used to
983 define two profiles, giving differential output. This is done by
984 enclosing each of the two specifications within curly braces, as shown
985 in the examples below. Any specifications outside of curly braces are
989 <sect2 id="profile-spec-examples">
990 <title>Examples</title>
993 Image summaries for all profiles with <constant>DATA_MEM_REFS</constant>
994 samples in the saved session called "stresstest" :
997 # opreport session:stresstest event:DATA_MEM_REFS
1001 Symbol summary for the application called "test_sym53c8xx,9xx". Note the
1002 escaping is necessary as <option>image:</option> takes a comma-separated list.
1005 # opreport -l ./test/test_sym53c8xx\,9xx
1009 Image summaries for all binaries in the <filename>test</filename> directory,
1010 excepting <filename>boring-test</filename> :
1013 # opreport image:./test/\* image-exclude:./test/boring-test
1017 Differential profile of a binary stored in two archives :
1020 # opreport -l /bin/bash { archive:./orig } { archive:./new }
1024 Differential profile of an archived binary with the current session :
1027 # opreport -l /bin/bash { archive:./orig } { }
1030 </sect2> <!-- profile spec examples -->
1032 <sect2 id="profile-spec-details">
1033 <title>Profile specification parameters</title>
1037 <term><option>archive:</option><emphasis>archivepath</emphasis></term>
1039 A path to an archive made with <command>oparchive</command>.
1040 Absence of this tag, unlike others, means "the current system",
1041 equivalent to specifying "archive:".
1045 <term><option>session:</option><emphasis>sessionlist</emphasis></term>
1047 A comma-separated list of session names to resolve in. Absence of this
1048 tag, unlike others, means "the current session", equivalent to
1049 specifying "session:current".
1053 <term><option>session-exclude:</option><emphasis>sessionlist</emphasis></term>
1055 A comma-separated list of sessions to exclude.
1059 <term><option>image:</option><emphasis>imagelist</emphasis></term>
1061 A comma-separated list of image names to resolve. Each entry may be relative
1062 path, <command>glob</command>-style name, or full path, e.g.</para>
1063 <screen>opreport 'image:/usr/bin/oprofiled,*op*,./opreport'</screen>
1068 <term><option>image-exclude:</option><emphasis>imagelist</emphasis></term>
1070 Same as <option>image:</option>, but the matching images are excluded.
1075 <term><option>lib-image:</option><emphasis>imagelist</emphasis></term>
1077 Same as <option>image:</option>, but only for images that are for
1078 a particular primary binary image (namely, an application).
1083 <term><option>lib-image-exclude:</option><emphasis>imagelist</emphasis></term>
1085 Same as <option>lib-image:</option>, but the matching images
1091 <term><option>event:</option><emphasis>eventlist</emphasis></term>
1093 The symbolic event name to match on, e.g. <option>event:DATA_MEM_REFS</option>.
1094 You can pass a list of events for side-by-side comparison with <command>opreport</command>.
1099 <term><option>count:</option><emphasis>eventcountlist</emphasis></term>
1101 The event count to match on, e.g. <option>event:DATA_MEM_REFS count:30000</option>.
1102 Note that this value refers to the count value in the event spec you passed
1103 to <command>operf</command> when setting up to do a
1104 profile run. It has nothing to do with the sample counts in the profile data
1106 You can pass a list of events for side-by-side comparison with <command>opreport</command>.
1111 <term><option>unit-mask:</option><emphasis>masklist</emphasis></term>
1113 The unit mask value of the event to match on, e.g. <option>unit-mask:1</option>.
1114 You can pass a list of events for side-by-side comparison with <command>opreport</command>.
1119 <term><option>cpu:</option><emphasis>cpulist</emphasis></term>
1121 Only consider profiles for the given numbered CPU (starting from zero).
1122 This is only useful when using CPU profile separation.
1127 <term><option>tgid:</option><emphasis>pidlist</emphasis></term>
1129 Only consider profiles for the given task groups. Unless some program
1130 is using threads, the task group ID of a process is the same
1131 as its process ID. This option corresponds to the POSIX
1132 notion of a thread group.
1133 This is only useful when using per-process profile separation.
1138 <term><option>tid:</option><emphasis>tidlist</emphasis></term>
1140 Only consider profiles for the given threads. When using
1141 recent thread libraries, all threads in a process share the
1142 same task group ID, but have different thread IDs. You can
1143 use this option in combination with <option>tgid:</option> to
1144 restrict the results to particular threads within a process.
1145 This is only useful when using per-process profile separation.
1152 <sect2 id="locating-and-managing-binary-images">
1153 <title>Locating and managing binary images</title>
1155 Each session's sample files can be found in the $SESSION_DIR/samples/ directory (default
1156 for <command>operf</command> is <filename><cur_dir>/oprofile_data/samples/</filename>).
1157 These are used, along with the binary image files, to produce human-readable data.
1158 In some circumstances (e.g., kernel modules), OProfile
1159 will not be able to find the binary images. All the tools have an <option>--image-path</option>
1160 option to which you can pass a comma-separated list of alternate paths to search. For example,
1161 I can let OProfile find my 2.6 modules by using <command>--image-path /lib/modules/2.6.0/kernel/</command>.
1162 It is your responsibility to ensure that the correct images are found when using this
1166 Note that if a binary image changes after the sample file was created, you won't be able to get useful
1167 symbol-based data out. This situation is detected for you. If you replace a binary, you should
1168 make sure to save the old binary if you need to do comparative profiles.
1173 <sect2 id="no-results">
1174 <title>What to do when you don't get any results</title>
1176 When attempting to get output, you may see the error :
1179 error: no sample files found: profile specification too strict ?
1182 What this is saying is that the profile specification you passed in,
1183 when matched against the available sample files, resulted in no matches.
1184 There are a number of reasons this might happen:
1187 <varlistentry><term>spelling</term><listitem><para>
1188 You specified a binary name, but spelt it wrongly. Check your spelling !
1189 </para></listitem></varlistentry>
1190 <varlistentry><term>profiler wasn't running</term><listitem><para>
1191 Make very sure that OProfile was actually up and running when you ran
1192 the application you wish to profile.
1193 </para></listitem></varlistentry>
1194 <varlistentry><term>application didn't run long enough</term><listitem><para>
1195 Remember OProfile is a statistical profiler - you're not guaranteed to
1196 get samples for short-running programs. You can help this by using a
1197 lower count for the performance counter, so there are a lot more samples
1199 </para></listitem></varlistentry>
1200 <varlistentry><term>application spent most of its time in libraries</term><listitem><para>
1201 Similarly, if the application spends little time in the main binary image
1202 itself, with most of it spent in shared libraries it uses, you might
1203 not see any samples for the binary image (i.e., executable) itself.
1204 </para></listitem></varlistentry>
1205 <varlistentry><term>specification was really too strict</term><listitem><para>
1206 For example, you specified something like <option>tgid:3433</option>,
1207 but no task with that group ID ever ran the code.
1208 </para></listitem></varlistentry>
1209 <varlistentry><term>application didn't generate any events</term><listitem><para>
1210 If you're profiling a particular event, for example counting MMX
1211 operations, the code might simply have not generated any events in the
1212 first place. Verify the code you're profiling does what you expect it
1214 </para></listitem></varlistentry>
1215 <varlistentry><term>you didn't specify kernel module name correctly</term><listitem><para>
1216 If you're trying to get reports for a kernel
1217 module, make sure to use the <option>-p</option> option, and specify the
1218 module name <emphasis>with</emphasis> the <filename>.ko</filename>
1219 extension. Check if the module is one loaded from initrd.
1220 </para></listitem></varlistentry>
1225 </sect1> <!-- profile-spec -->
1227 <sect1 id="opreport">
1228 <title>Image summaries and symbol summaries (<command>opreport</command>)</title>
1230 The <command>opreport</command> utility is the primary utility you will use for
1231 getting formatted data out of OProfile. It produces two types of data: image summaries
1232 and symbol summaries. An image summary lists the number of samples for individual
1233 binary images such as libraries or applications. Symbol summaries provide per-symbol
1234 profile data. In the following truncated example, we see an image summary for the whole
1238 $ opreport --long-filenames
1239 CPU: Intel Sandy Bridge microarchitecture, speed 2401 MHz (estimated)
1240 Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (No unit mask) count 100000
1244 22577 28.9011 /usr/bin/Xorg
1248 16846 74.6158 /proc/kallsyms
1249 2126 9.4167 /usr/bin/Xorg
1250 763 3.3795 /usr/lib64/libpixman-1.so.0.26.2
1252 17402 22.2766 /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.55.x86_64/jre/bin/java
1256 5666 32.5595 anon (tgid:29664 range:0x7f3475000000-0x7f347616ffff)
1257 2312 13.2858 /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.55.x86_64/jre/lib/amd64/server/libjvm.so
1259 11554 14.7904 /home/user1/oprof-install/bin/operf
1263 7467 64.6270 /proc/kallsyms
1264 1691 14.6356 /usr/bin/operf
1265 1324 11.4592 /lib64/libc-2.12.so
1266 455 3.9380 /usr/lib64/libstdc++.so.6.0.13
1272 If we had specified <option>--symbols</option> in the previous command, we would have
1273 gotten a symbol summary of all the images across the entire system. We can restrict this to only
1274 part of the system profile; for example,
1275 below is a symbol summary for the <command>operf</command> program used to collect the profile.
1278 $ opreport -l -p /lib/modules/`uname -r` `which operf` 2>/dev/null | more
1279 CPU: Intel Sandy Bridge microarchitecture, speed 2401 MHz (estimated)
1280 Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (No unit mask) count 100000
1281 samples % image name symbol name
1282 860 7.4607 kallsyms avtab_search_node
1283 474 4.1121 operf OP_perf_utils::op_write_event(event_union*, unsigned long long)
1284 461 3.9993 kallsyms avc_has_perm_noaudit
1285 455 3.9473 libstdc++.so.6.0.13 /usr/lib64/libstdc++.so.6.0.13
1286 412 3.5742 libc-2.12.so _IO_vfscanf
1287 369 3.2012 kallsyms __d_lookup
1288 350 3.0363 kallsyms sidtab_context_to_sid
1289 274 2.3770 operf OP_perf_utils::op_record_process_exec_mmaps(int, int, int, operf_record*)
1290 232 2.0127 operf operf_process_info::find_mapping_for_sample(unsigned long long, bool)
1291 222 1.9259 kallsyms __link_path_walk
1292 191 1.6570 kallsyms pipe_read
1293 34 0.2950 ext4.ko ext4_mark_iloc_dirty
1298 These are the two basic ways you are most likely to use regularly, but <command>opreport</command>
1299 can do a lot more than that, as described below.
1302 <sect2 id="opreport-merging">
1303 <title>Merging separate profiles</title>
1305 If you have used one of the <option>--separate[*]</option> options
1306 whilst profiling, there can be several separate profiles for
1307 a single binary image within a session. Normally the output
1308 will keep these images separated. So, for example, if you profiled
1309 with separation on a per-cpu basis (<code>operf --separate-cpu</code>),
1310 you would see separate columns in
1311 the output of <command>opreport</command> for each CPU where samples
1312 were recorded. But it can be useful to merge these results back together
1313 to make the report more readable. The <option>--merge</option> option allows
1317 <sect2 id="opreport-comparison">
1318 <title>Side-by-side multiple results</title>
1319 If you have used multiple events when profiling, by default you get
1320 side-by-side results of each event's sample values from <command>opreport</command>.
1321 You can restrict which events to list by appropriate use of the
1322 <option>event:</option> profile specifications, etc.
1325 <sect2 id="opreport-callgraph">
1326 <title>Callgraph output</title>
1328 This section provides details on how to use the OProfile callgraph feature.
1331 <title>Callgraph details</title>
1333 When using the <option>--callgraph</option> option, you can see what
1334 functions are calling other functions in the output. Consider the
1338 #include <string.h>
1339 #include <stdlib.h>
1340 #include <stdio.h>
1344 static int compare(const void *s1, const void *s2)
1346 return strcmp(s1, s2);
1349 static void repeat(void)
1352 char *strings[SIZE];
1353 char str[] = "abcdefghijklmnopqrstuvwxyz";
1355 for (i = 0; i < SIZE; ++i) {
1356 strings[i] = strdup(str);
1360 qsort(strings, SIZE, sizeof(char *), compare);
1370 When running with the call-graph option, OProfile will
1371 record the function stack every time it takes a sample.
1372 <command>opreport --callgraph</command> outputs an entry for each
1373 function, where each entry looks similar to:
1376 samples % image name symbol name
1378 127036 99.8452 cg repeat
1379 84590 42.5084 libc-2.3.2.so strfry
1380 84590 66.4838 libc-2.3.2.so strfry [self]
1381 39169 30.7850 libc-2.3.2.so random_r
1382 3475 2.7312 libc-2.3.2.so __i686.get_pc_thunk.bx
1383 -------------------------------------------------------------------------------
1386 Here the non-indented line is the function we're focussing upon
1387 (<function>strfry()</function>). This
1388 line is the same as you'd get from a normal <command>opreport</command>
1392 Above the non-indented line we find the functions that called this
1393 function (for example, <function>repeat()</function> calls
1394 <function>strfry()</function>). The samples and percentage values here
1395 refer to the number of times we took a sample where this call was found
1396 in the stack; the percentage is relative to all other callers of the
1397 function we're focussing on. Note that these values are
1398 <emphasis>not</emphasis> call counts; they only reflect the call stack
1399 every time a sample is taken; that is, if a call is found in the stack
1400 at the time of a sample, it is recorded in this count.
1403 Below the line are functions that are called by
1404 <function>strfry()</function> (called <emphasis>callees</emphasis>).
1405 It's clear here that <function>strfry()</function> calls
1406 <function>random_r()</function>. We also see a special entry with a
1407 "[self]" marker. This records the normal samples for the function, but
1408 the percentage becomes relative to all callees. This allows you to
1409 compare time spent in the function itself compared to functions it
1410 calls. Note that if a function calls itself, then it will appear in the
1411 list of callees of itself, but without the "[self]" marker; so recursive
1412 calls are still clearly separable.
1415 You may have noticed that the output lists <function>main()</function>
1416 as calling <function>strfry()</function>, but it's clear from the source
1417 that this doesn't actually happen. See <xref
1418 linkend="interpreting-callgraph" /> for an explanation.
1421 <sect3 id="cg-with-jitsupport">
1422 <title>Callgraph is not supported with JIT samples</title>
1424 Callgraph output where anonymously mapped code is in the callstack can sometimes be misleading.
1425 For all such code, the samples for the anonymously mapped code are stored in a samples subdirectory
1426 named <filename>{anon:anon}/<tgid>.<begin_addr>.<end_addr></filename>.
1427 As stated earlier, if this anonymously mapped code is JITed code from a supported VM like Java,
1428 OProfile creates an ELF file to provide a (somewhat) permanent backing file for the code.
1429 However, when viewing callgraph output, any anonymously mapped code in the callstack
1430 will be attributed to <filename>anon (<tgid>: range:<begin_addr>-<end_addr></filename>,
1431 even if a <filename>.jo</filename> ELF file had been created for it. See the example below.
1434 -------------------------------------------------------------------------------
1435 1 2.2727 libj9ute23.so java.bin traceV
1436 2 4.5455 libj9ute23.so java.bin utsTraceV
1437 4 9.0909 libj9trc23.so java.bin fillInUTInterfaces
1438 37 84.0909 libj9trc23.so java.bin twGetSequenceCounter
1439 8 0.0154 libj9prt23.so java.bin j9time_hires_clock
1440 27 61.3636 anon (tgid:10014 range:0x100000-0x103000) java.bin (no symbols)
1441 9 20.4545 libc-2.4.so java.bin gettimeofday
1442 8 18.1818 libj9prt23.so java.bin j9time_hires_clock [self]
1443 -------------------------------------------------------------------------------
1446 The output shows that "anon (tgid:10014 range:0x100000-0x103000)" was a callee of
1447 <code>j9time_hires_clock</code>, even though the ELF file <filename>10014.jo</filename> was
1448 created for this profile run. Unfortunately, there is currently no way to correlate
1449 that anonymous callgraph entry with its corresponding <filename>.jo</filename> file.
1454 </sect2> <!-- opreport-callgraph -->
1456 <sect2 id="opreport-diff">
1457 <title>Differential profiles with <command>opreport</command></title>
1460 Often, we'd like to be able to compare two profiles. For example, when
1461 analysing the performance of an application, we'd like to make code
1462 changes and examine the effect of the change. This is supported in
1463 <command>opreport</command> by giving a profile specification that
1464 identifies two different profiles. The general form is of:
1467 $ opreport <shared-spec> { <first-profile> } { <second-profile> }
1470 We lost our Dragon book down the back of the sofa, so you have to be
1471 careful to have spaces around those braces, or things will get
1472 hopelessly confused. We can only apologise.
1475 For each of the profiles, the shared section is prefixed, and then the
1476 specification is analysed. The usual parameters work both within the
1477 shared section, and in the sub-specification within the curly braces.
1480 A typical way to use this feature is with archives created with
1481 <command>oparchive</command>. Let's look at an example:
1485 $ oparchive -o orig ./a
1486 # edit and recompile a
1488 # now compare the current profile of a with the archived profile
1489 $ opreport --session-dir=`pwd`/oprofile_data/ -xl ./a { archive:./orig } { }
1490 CPU: PIII, speed 863.233 MHz (estimated)
1491 Counted CPU_CLK_UNHALTED events (clocks processor is not halted) with a
1492 unit mask of 0x00 (No unit mask) count 100000
1493 samples % diff % symbol name
1494 92435 48.5366 +0.4999 a
1497 48787 25.6175 -2.2e-01 b
1500 Note that we specified an empty second profile in the curly braces, as
1501 we wanted to use the current session; alternatively, we could
1502 have specified another archive, or a tgid etc. We specified the binary
1503 <command>a</command> in the shared section, so we matched that in both
1504 the profiles we're diffing.
1507 As in the normal output, the results are sorted by the number of
1508 samples, and the percentage field represents the relative percentage of
1509 the symbol's samples in the second profile.
1512 Notice the new column in the output. This value represents the
1513 percentage change of the relative percent between the first and the
1514 second profile: roughly, "how much more important this symbol is".
1515 Looking at the symbol <function>a()</function>, we can see that it took
1516 roughly the same amount of the total profile in both the first and the
1517 second profile. The function <function>c()</function> was not in the new
1518 profile, so has been marked with <function>---</function>. Note that the
1519 sample value is the number of samples in the first profile; since we're
1520 displaying results for the second profile, we don't list a percentage
1521 value for it, as it would be meaningless. <function>d()</function> is
1522 new in the second profile, and consequently marked with
1523 <function>+++</function>.
1526 When comparing profiles between different binaries, it should be clear
1527 that functions can change in terms of VMA and size. To avoid this
1528 problem, <command>opreport</command> considers a symbol to be the same
1529 if the symbol name, image name, and owning application name all match;
1530 any other factors are ignored. Note that the check for application name
1531 means that trying to compare library profiles between two different
1532 applications will not work as you might expect: each symbol will be
1533 considered different.
1536 </sect2> <!-- opreport-diff -->
1538 <sect2 id="opreport-anon">
1539 <title>Anonymous executable mappings</title>
1541 Many applications, typically ones involving dynamic compilation into
1542 machine code (just-in-time, or "JIT", compilation), have executable mappings that
1543 are not backed by an ELF file. <command>opreport</command> has basic support for showing the
1544 samples taken in these regions; for example:
1546 $ opreport /usr/bin/mono -l
1547 CPU: ppc64 POWER5, speed 1654.34 MHz (estimated)
1548 Counted CYCLES events (Processor Cycles using continuous sampling) with a unit mask of 0x00 (No unit mask) count 100000
1549 samples % image name symbol name
1550 47 58.7500 mono (no symbols)
1551 14 17.5000 anon (tgid:3189 range:0xf72aa000-0xf72fa000) (no symbols)
1552 9 11.2500 anon (tgid:3189 range:0xf6cca000-0xf6dd9000) (no symbols)
1557 Note that, since such mappings are dependent upon individual invocations of
1558 a binary, these mappings are always listed as a dependent image.
1559 Equally, the results are not affected by the <option>--merge</option>
1563 As shown in the opreport output above, OProfile is unable to attribute the samples to any
1564 symbol(s) because there is no ELF file for this code.
1565 Enhanced support for JITed code is now available for some virtual machines;
1566 e.g., the Java Virtual Machine. For details about OProfile output for
1567 JITed code, see <xref linkend="getting-jit-reports" />.
1569 <para>For more information about JIT support in OProfile, see <xref linkend="jitsupport"/>.
1571 </sect2> <!-- opreport-anon -->
1573 <sect2 id="opreport-xml">
1574 <title>XML formatted output</title>
1576 The --xml option can be used to generate XML instead of the usual
1577 text format. This allows opreport to eliminate some of the constraints
1578 dictated by the two dimensional text format. For example, it is possible
1579 to separate the sample data across multiple events, cpus and threads. The XML
1580 schema implemented by opreport is found in doc/opreport.xsd. It contains
1581 more detailed comments about the structure of the XML generated by opreport.
1584 Since XML is consumed by a client program rather than a user, its structure
1585 is fairly static. In particular, the --sort option is incompatible with the
1586 --xml option. Percentages are not dislayed in the XML so the options related
1587 to percentages will have no effect. Full pathnames are always displayed in
1588 the XML so --long-filenames is not necessary. The --details option will cause
1589 all of the individual sample data to be included in the XML as well as the
1590 instruction byte stream for each symbol (for doing disassembly) and can result
1591 in very large XML files.
1593 </sect2> <!-- opreport-xml -->
1595 <sect2 id="opreport-options">
1596 <title>Options for <command>opreport</command></title>
1599 <varlistentry><term><option>--accumulated / -a</option></term><listitem><para>
1600 Accumulate sample and percentage counts in the symbol list.
1601 </para></listitem></varlistentry>
1602 <varlistentry><term><option>--callgraph / -c</option></term><listitem><para>
1603 Show callgraph information.
1604 </para></listitem></varlistentry>
1605 <varlistentry><term><option>--debug-info / -g</option></term><listitem><para>
1606 Show source file and line for each symbol.
1607 </para></listitem></varlistentry>
1608 <varlistentry><term><option>--demangle / -D none|normal|smart</option></term><listitem><para>
1609 none: no demangling. normal: use default demangler (default) smart: use
1610 pattern-matching to make C++ symbol demangling more readable.
1611 </para></listitem></varlistentry>
1612 <varlistentry><term><option>--details / -d</option></term><listitem><para>
1613 Show per-instruction details for all selected symbols. Note that, for
1614 binaries without symbol information, the VMA values shown are raw file
1615 offsets for the image binary.
1616 </para></listitem></varlistentry>
1617 <varlistentry><term><option>--exclude-dependent / -x</option></term><listitem><para>
1618 Do not include application-specific images for libraries, kernel modules
1620 </para></listitem></varlistentry>
1621 <varlistentry><term><option>--exclude-symbols / -e [symbols]</option></term><listitem><para>
1622 Exclude all the symbols in the given comma-separated list.
1623 </para></listitem></varlistentry>
1624 <varlistentry><term><option>--global-percent / -%</option></term><listitem><para>
1625 Make all percentages relative to the whole profile.
1626 </para></listitem></varlistentry>
1627 <varlistentry><term><option>--help / -? / --usage</option></term><listitem><para>
1629 </para></listitem></varlistentry>
1630 <varlistentry><term><option>--image-path / -p [paths]</option></term><listitem><para>
1631 Comma-separated list of additional paths to search for binaries.
1632 This is needed to find kernel modules.
1633 </para></listitem></varlistentry>
1634 <varlistentry><term><option>--root / -R [path]</option></term><listitem><para>
1635 A path to a filesystem to search for additional binaries.
1636 </para></listitem></varlistentry>
1637 <varlistentry><term><option>--include-symbols / -i [symbols]</option></term><listitem><para>
1638 Only include symbols in the given comma-separated list.
1639 </para></listitem></varlistentry>
1640 <varlistentry><term><option>--long-filenames / -f</option></term><listitem><para>
1641 Output full paths instead of basenames.
1642 </para></listitem></varlistentry>
1643 <varlistentry><term><option>--merge / -m [lib,cpu,tid,tgid,unitmask,all]</option></term><listitem><para>
1644 Merge any profiles separated in a --separate session.
1645 </para></listitem></varlistentry>
1646 <varlistentry><term><option>--no-header</option></term><listitem><para>
1647 Don't output a header detailing profiling parameters.
1648 </para></listitem></varlistentry>
1649 <varlistentry><term><option>--output-file / -o [file]</option></term><listitem><para>
1650 Output to the given file instead of stdout.
1651 </para></listitem></varlistentry>
1652 <varlistentry><term><option>--reverse-sort / -r</option></term><listitem><para>
1653 Reverse the sort from the default.
1654 </para></listitem></varlistentry>
1655 <varlistentry><term><option>--session-dir=dir_path</option></term><listitem><para>
1656 Use sample database from the specified directory <filename>dir_path</filename> instead
1657 of the default location. If this option is not specified, then opreport will search for
1658 samples in <filename><cur_dir>/oprofile_data</filename>
1659 first. If that directory does not exist, the standard session-dir of
1660 <filename>/var/lib/oprofile</filename> is used
1661 as the session directory.
1662 </para></listitem></varlistentry>
1663 <varlistentry><term><option>--show-address / -w</option></term><listitem><para>
1664 Show the VMA address of each symbol (off by default).
1665 </para></listitem></varlistentry>
1666 <varlistentry><term><option>--sort / -s [vma,sample,symbol,debug,image]</option></term><listitem><para>
1667 Sort the list of symbols by, respectively, symbol address,
1668 number of samples, symbol name, debug filename and line number,
1669 binary image filename.
1670 </para></listitem></varlistentry>
1671 <varlistentry><term><option>--symbols / -l</option></term><listitem><para>
1672 List per-symbol information instead of a binary image summary.
1673 </para></listitem></varlistentry>
1674 <varlistentry><term><option>--threshold / -t [percentage]</option></term><listitem><para>
1675 Only output data for symbols that have more than the given percentage
1676 of total samples. For profiles using multiple events, if the threshold is reached
1677 for any event, then all sample data for the symbol is shown.
1678 </para></listitem></varlistentry>
1679 <varlistentry><term><option>--verbose / -V [options]</option></term><listitem><para>
1680 Give verbose debugging output.
1681 </para></listitem></varlistentry>
1682 <varlistentry><term><option>--version / -v</option></term><listitem><para>
1684 </para></listitem></varlistentry>
1685 <varlistentry><term><option>--xml / -X</option></term><listitem><para>
1686 Generate XML output.
1687 </para></listitem></varlistentry>
1692 </sect1> <!-- opreport -->
1694 <sect1 id="opannotate">
1695 <title>Outputting annotated source (<command>opannotate</command>)</title>
1697 The <command>opannotate</command> utility generates annotated source files or assembly listings, optionally
1699 If you want to see the source file, the profiled application needs to have debug information, and the source
1700 must be available through this debug information. For GCC, you must use the <option>-g</option> option
1701 when you are compiling.
1702 If the binary doesn't contain sufficient debug information, you can still
1703 use <command>opannotate <option>--assembly</option></command> to get annotated assembly
1704 as long as the binary has (at least) symbol information.
1707 Note that for the reason explained in <xref linkend="hardware-counters" /> the results can be
1708 inaccurate. The debug information itself can add other problems; for example, the line number for a symbol can be
1709 incorrect. Assembly instructions can be re-ordered and moved by the compiler, and this can lead to
1710 crediting source lines with samples not really "owned" by this line. Also see
1711 <xref linkend="interpreting" />.
1714 You can output the annotation to one single file, containing all the source found using the
1715 <option>--source</option>. You can use this in conjunction with <option>--assembly</option>
1716 to get combined source/assembly output.
1719 You can also output a directory of annotated source files that maintains the structure of
1720 the original sources. Each line in the annotated source is prepended with the samples
1721 for that line. Additionally, each symbol is annotated giving details for the symbol
1722 as a whole. An example:
1725 $ opannotate --source --output-dir=annotated /usr/local/oprofile-pp/bin/oprofiled
1726 $ ls annotated/home/moz/src/oprofile-pp/daemon/
1727 opd_cookie.h opd_image.c opd_kernel.c opd_sample_files.c oprofiled.c
1730 Line numbers are maintained in the source files, but each file has
1731 a footer appended describing the profiling details. The actual annotation
1732 looks something like this :
1736 :static uint64_t pop_buffer_value(struct transient * trans)
1737 11510 1.9661 :{ /* pop_buffer_value total: 89901 15.3566 */
1740 10227 1.7469 : if (!trans->remaining) {
1741 : fprintf(stderr, "BUG: popping empty buffer !\n");
1742 : exit(EXIT_FAILURE);
1745 : val = get_buffer_value(trans->buffer, 0);
1746 2281 0.3896 : trans->remaining--;
1747 2296 0.3922 : trans->buffer += kernel_pointer_size;
1754 The first number on each line is the number of samples, whilst the second is
1755 the relative percentage of total samples.
1758 <sect2 id="opannotate-finding-source">
1759 <title>Locating source files</title>
1761 Of course, <command>opannotate</command> needs to be able to locate the source files
1762 for the binary image(s) in order to produce output. Some binary images have debug
1763 information where the given source file paths are relative, not absolute. You can
1764 specify search paths to look for these files (similar to <command>gdb</command>'s
1765 <option>dir</option> command) with the <option>--search-dirs</option> option.
1768 Sometimes you may have a binary image which gives absolute paths for the source files,
1769 but you have the actual sources elsewhere (commonly, you've installed an SRPM for
1770 a binary on your system and you want annotation from an existing profile). You can
1771 use the <option>--base-dirs</option> option to redirect OProfile to look somewhere
1772 else for source files. For example, imagine we have a binary generated from a source
1773 file that is given in the debug information as <filename>/tmp/build/libfoo/foo.c</filename>,
1774 and you have the source tree matching that binary installed in <filename>/home/user/libfoo/</filename>.
1775 You can redirect OProfile to find <filename>foo.c</filename> correctly like this :
1778 $ opannotate --source --base-dirs=/tmp/build/libfoo/ --search-dirs=/home/user/libfoo/ --output-dir=annotated/ /lib/libfoo.so
1781 You can specify multiple (comma-separated) paths to both options.
1785 <sect2 id="opannotate-details">
1786 <title>Usage of <command>opannotate</command></title>
1789 <varlistentry><term><option>--assembly / -a</option></term><listitem><para>
1790 Output annotated assembly. If this is combined with --source, then mixed
1791 source / assembly annotations are output.
1792 </para></listitem></varlistentry>
1793 <varlistentry><term><option>--base-dirs / -b [paths]/</option></term><listitem><para>
1794 Comma-separated list of path prefixes. This can be used to point OProfile to a
1795 different location for source files when the debug information specifies an
1796 absolute path on your system for the source that does not exist. The prefix
1797 is stripped from the debug source file paths, then searched in the search dirs
1798 specified by <option>--search-dirs</option>.
1799 </para></listitem></varlistentry>
1800 <varlistentry><term><option>--demangle / -D none|normal|smart</option></term><listitem><para>
1801 none: no demangling. normal: use default demangler (default) smart: use
1802 pattern-matching to make C++ symbol demangling more readable.
1803 </para></listitem></varlistentry>
1804 <varlistentry><term><option>--exclude-dependent / -x</option></term><listitem><para>
1805 Do not include application-specific images for libraries, kernel modules
1807 </para></listitem></varlistentry>
1808 <varlistentry><term><option>--exclude-file [files]</option></term><listitem><para>
1809 Exclude all files in the given comma-separated list of glob patterns.
1810 This option is supported solely with the <code>--source</code>
1811 option. It can be used to filter out source files in the output using the
1812 following types of specifications:
1814 <listitem>filenames (basename -- i.e., no path)</listitem>
1815 <listitem>filename glob specifications (all files whose base filename matches the given pattern)</listitem>
1816 <listitem>directory segments (all source files located in the specified directory; e.g. "libio")</listitem>
1817 <listitem>directory segment glob specifications (e.g., "libi*")</listitem>
1819 </para></listitem></varlistentry>
1820 <varlistentry><term><option>--exclude-symbols / -e [symbols]</option></term><listitem><para>
1821 Exclude all the symbols in the given comma-separated list.
1822 </para></listitem></varlistentry>
1823 <varlistentry><term><option>--help / -? / --usage</option></term><listitem><para>
1825 </para></listitem></varlistentry>
1826 <varlistentry><term><option>--image-path / -p [paths]</option></term><listitem><para>
1827 Comma-separated list of additional paths to search for binaries.
1828 This is needed to find kernel modules.
1829 </para></listitem></varlistentry>
1830 <varlistentry><term><option>--root / -R [path]</option></term><listitem><para>
1831 A path to a filesystem to search for additional binaries.
1832 </para></listitem></varlistentry>
1833 <varlistentry><term><option>--include-file [files]</option></term><listitem><para>
1834 Only include files in the given comma-separated list of glob patterns.
1835 The same rules apply for this option as for the <code>--exclude-file</code> option.
1836 </para></listitem></varlistentry>
1837 <varlistentry><term><option>--include-symbols / -i [symbols]</option></term><listitem><para>
1838 Only include symbols in the given comma-separated list.
1839 </para></listitem></varlistentry>
1840 <varlistentry><term><option>--objdump-params [params]</option></term><listitem><para>
1841 Pass the given parameters as extra values when calling objdump.
1842 If more than one option is to be passed to objdump, the parameters must be enclosed in a
1846 An example of where this option is useful is when your toolchain does not
1847 automatically recognize instructions that are specific to your processor.
1848 For example, on IBM POWER7/RHEL 6, objdump must be told that a binary file may have
1849 POWER7-specific instructions. The <command>opannotate</command> option to show the POWER7-specific
1852 --objdump-params=-Mpower7
1856 The <command>opannotate</command> option to show the POWER7-specific instructions,
1857 the source code (--source) and the line numbers (-l) would be:
1859 --objdump-params="-Mpower7 -l --source"
1861 </para></listitem></varlistentry>
1862 <varlistentry><term><option>--output-dir / -o [dir]</option></term><listitem><para>
1863 Output directory. This makes opannotate output one annotated file for each
1864 source file. This option can't be used in conjunction with --assembly.
1865 </para></listitem></varlistentry>
1866 <varlistentry><term><option>--search-dirs / -d [paths]</option></term><listitem><para>
1867 Comma-separated list of paths to search for source files. This is useful to find
1868 source files when the debug information only contains relative paths.
1869 </para></listitem></varlistentry>
1870 <varlistentry><term><option>--source / -s</option></term><listitem><para>
1871 Output annotated source. This requires debugging information to be available
1873 </para></listitem></varlistentry>
1874 <varlistentry><term><option>--session-dir=dir_path</option></term><listitem><para>
1875 Use sample database from the specified directory <filename>dir_path</filename> instead
1876 of the default location. If this option is not specified, then opannotate will search for
1877 samples in <filename><cur_dir>/oprofile_data</filename>
1878 first. If that directory does not exist, the standard session-dir of
1879 <filename>/var/lib/oprofile</filename> is used
1880 as the session directory.
1881 </para></listitem></varlistentry>
1882 <varlistentry><term><option>--threshold / -t [percentage]</option></term><listitem><para>
1883 For annotated assembly, only output data for symbols that have more than the given percentage
1884 of total samples. For profiles using multiple events, if the threshold is reached
1885 for any event, then all sample data for the symbol is shown.
1888 For annotated source, only output data for source files that have more than the given percentage
1889 of total samples. For profiles using multiple events, if the threshold is reached
1890 for any event, then all sample data for the source file is shown.
1891 </para></listitem></varlistentry>
1892 <varlistentry><term><option>--verbose / -V [options]</option></term><listitem><para>
1893 Give verbose debugging output.
1894 </para></listitem></varlistentry>
1895 <varlistentry><term><option>--version / -v</option></term><listitem><para>
1897 </para></listitem></varlistentry>
1901 </sect2> <!-- opannotate-details -->
1903 </sect1> <!-- opannotate -->
1905 <sect1 id="getting-jit-reports">
1906 <title>OProfile results with JIT samples</title>
1908 After profiling a Java (or other supported VM) application, the
1909 OProfile JIT support creates ELF binaries from the
1910 intermediate files that were written by the agent library.
1911 The ELF binaries are named <filename><tgid>.jo</filename>.
1912 With the symbol information stored in these ELF files, it is
1913 possible to map samples to the appropriate symbols.
1916 The usual analysis tools (<command>opreport</command> and/or
1917 <command>opannotate</command>) can now be used
1918 to get symbols and assembly code for the instrumented VM processes.
1921 Below is an example of a profile report of a Java application that has been
1922 instrumented with the provided agent library.
1924 $ opreport -l /usr/lib/jvm/jre-1.5.0-ibm/bin/java
1925 CPU: Core Solo / Duo, speed 2167 MHz (estimated)
1926 Counted CPU_CLK_UNHALTED events (Unhalted clock cycles) with a unit mask of 0x00 (Unhalted core cycles) count 100000
1927 samples % image name symbol name
1928 186020 50.0523 no-vmlinux no-vmlinux (no symbols)
1929 34333 9.2380 7635.jo java void test.f1()
1930 19022 5.1182 libc-2.5.so libc-2.5.so _IO_file_xsputn@@GLIBC_2.1
1931 18762 5.0483 libc-2.5.so libc-2.5.so vfprintf
1932 16408 4.4149 7635.jo java void test$HelloThread.run()
1933 16250 4.3724 7635.jo java void test$test_1.f2(int)
1934 15303 4.1176 7635.jo java void test.f2(int, int)
1935 13252 3.5657 7635.jo java void test.f2(int)
1936 5165 1.3897 7635.jo java void test.f4()
1937 955 0.2570 7635.jo java void test$HelloThread.run()~
1942 Depending on the JVM that is used, certain options of opreport and opannotate
1943 do NOT work since they rely on debug information (e.g. source code line number)
1944 that is not always available. The Sun JVM does provide the necessary debug
1945 information via the JVMTI[PI] interface,
1946 but other JVMs do not.
1949 As you can see in the opreport output, the JIT support agent for Java
1950 generates symbols to include the class and method signature.
1951 A symbol with the suffix ˜<n> (e.g.
1952 <code>void test$HelloThread.run()˜1</code>) means that this is
1953 the <n>th occurrence of the identical name. This happens if a method is re-JITed.
1954 A symbol with the suffix %<n>, means that the address space of this symbol
1955 was reused during the sample session (see <xref linkend="overlapping-symbols" />).
1956 The value <n> is the percentage of time that this symbol/code was present in
1957 relation to the total lifetime of all overlapping other symbols. A symbol of the form
1958 <code><return_val> <class_name>$<method_sig></code> denotes an
1963 <sect1 id="opgprof">
1964 <title><command>gprof</command>-compatible output (<command>opgprof</command>)</title>
1966 If you're familiar with the output produced by <command>GNU gprof</command>,
1967 you may find <command>opgprof</command> useful. It takes a single binary
1968 as an argument, and produces a <filename>gmon.out</filename> file for use
1969 with <command>gprof -p</command>. If call-graph profiling is enabled,
1970 then this is also included.
1973 $ opgprof `which oprofiled` # generates gmon.out file
1974 $ gprof -p `which oprofiled` | head
1977 Each sample counts as 1 samples.
1978 % cumulative self self total
1979 time samples samples calls T1/call T1/call name
1980 33.13 206237.00 206237.00 odb_insert
1981 22.67 347386.00 141149.00 pop_buffer_value
1982 9.56 406881.00 59495.00 opd_put_sample
1983 7.34 452599.00 45718.00 opd_find_image
1984 7.19 497327.00 44728.00 opd_process_samples
1987 <sect2 id="opgprof-details">
1988 <title>Usage of <command>opgprof</command></title>
1991 <varlistentry><term><option>--help / -? / --usage</option></term><listitem><para>
1993 </para></listitem></varlistentry>
1994 <varlistentry><term><option>--image-path / -p [paths]</option></term><listitem><para>
1995 Comma-separated list of additional paths to search for binaries.
1996 This is needed to find kernel modules.
1997 </para></listitem></varlistentry>
1998 <varlistentry><term><option>--root / -R [path]</option></term><listitem><para>
1999 A path to a filesystem to search for additional binaries.
2000 </para></listitem></varlistentry>
2001 <varlistentry><term><option>--output-filename / -o [file]</option></term><listitem><para>
2002 Output to the given file instead of the default, gmon.out
2003 </para></listitem></varlistentry>
2004 <varlistentry><term><option>--threshold / -t [percentage]</option></term><listitem><para>
2005 Only output data for symbols that have more than the given percentage
2007 </para></listitem></varlistentry>
2008 <varlistentry><term><option>--verbose / -V [options]</option></term><listitem><para>
2009 Give verbose debugging output.
2010 <varlistentry><term><option>--session-dir=dir_path</option></term><listitem><para>
2011 Use sample database from the specified directory <filename>dir_path</filename> instead
2012 of the default location. If this option is not specified, then opgprof will search for
2013 samples in <filename><cur_dir>/oprofile_data</filename>
2014 first. If that directory does not exist, the standard session-dir of
2015 <filename>/var/lib/oprofile</filename> is used
2016 as the session directory.
2017 </para></listitem></varlistentry>
2018 </para></listitem></varlistentry>
2019 <varlistentry><term><option>--version / -v</option></term><listitem><para>
2021 </para></listitem></varlistentry>
2024 </sect2> <!-- opgprof-details -->
2026 </sect1> <!-- opgprof -->
2028 <sect1 id="oparchive">
2029 <title>Analyzing profile data on another system (<command>oparchive</command>)</title>
2031 The <command>oparchive</command> utility generates a directory populated
2032 with executable, debug, and oprofile sample files. This directory can be
2033 copied to another (host) machine and analyzed offline, with no further need to
2034 access the data collection machine (target).
2038 The following command, executed on the target system, will collect the
2039 sample files, the executables associated with the sample files, and the
2040 debuginfo files associated with the executables and copy them into
2041 <filename>/tmp/current_data</filename>:
2045 # oparchive -o /tmp/current_data
2049 When transferring archived profile data to a host machine for offline analysis,
2050 you need to determine if the oprofile ABI format is compatible between the
2051 target system and the host system; if it isn't, you must run the <command>opimport</command>
2052 command to convert the target's sample data files to the format of your host system.
2053 See <xref linkend="opimport"/> for more details.
2057 After your profile data is transferred to the host system and (if necessary)
2058 you have run the <command>opimport</command> command to convert the file
2059 format, you can now run the <command>opreport</command> and
2060 <command>opannotate</command> commands. However, you must provide an
2061 "archive specification" to let these post-processing tools know where to find
2062 of the profile data (sample files, executables, etc.); for example:
2066 # opreport archive:/home/user1/my_oprofile_archive --symbols
2070 Furthermore, if your profile was collected on your target system into a session-dir
2071 other than <filename>/var/lib/oprofile</filename>, the <command>oparchive</command>
2072 command will display a message similar to the following:
2076 # NOTE: The sample data in this archive is located at /home/user1/test-stuff/oprofile_data
2077 instead of the standard location of /var/lib/oprofile. Hence, when using opreport
2078 and other post-processing tools on this archive, you must pass the following option:
2079 --session-dir=/home/user1/test-stuff/oprofile_data
2083 Then the above <command>opreport</command> example would have to include that
2084 <option>--session-dir</option> option.
2089 In some host/target development environments, all target executables, libraries, and
2090 debuginfo files are stored in a root directory on the host to facilitate offline
2091 analysis. In such cases, the <command>oparchive</command> command collects more data
2092 than is necessary; so, when copying the resulting output of <command>oparchive</command>,
2093 you can skip all of the executables, etc, and just archive the <filename>$SESSION_DIR</filename>
2094 tree located within the output directory you specified in your <command>oparchive</command>
2095 command. Then, when running the <command>opreport</command> or <command>opannotate</command>
2096 commands on your host system, pass the <option>--root</option> option to point to the
2097 location of your target's executables, etc.
2101 <sect2 id="oparchive-details">
2102 <title>Usage of <command>oparchive</command></title>
2105 <varlistentry><term><option>--help / -? / --usage</option></term><listitem><para>
2107 </para></listitem></varlistentry>
2108 <varlistentry><term><option>--exclude-dependent / -x</option></term><listitem><para>
2109 Do not include application-specific images for libraries, kernel modules
2111 </para></listitem></varlistentry>
2112 <varlistentry><term><option>--image-path / -p [paths]</option></term><listitem><para>
2113 Comma-separated list of additional paths to search for binaries.
2114 This is needed to find kernel modules.
2115 </para></listitem></varlistentry>
2116 <varlistentry><term><option>--root / -R [path]</option></term><listitem><para>
2117 A path to a filesystem to search for additional binaries.
2118 </para></listitem></varlistentry>
2119 <varlistentry><term><option>--output-directory / -o [directory]</option></term><listitem><para>
2120 Output to the given directory. There is no default. This must be specified.
2121 </para></listitem></varlistentry>
2122 <varlistentry><term><option>--list-files / -l</option></term><listitem><para>
2123 Only list the files that would be archived, don't copy them.
2124 </para></listitem></varlistentry>
2125 <varlistentry><term><option>--verbose / -V [options]</option></term><listitem><para>
2126 Give verbose debugging output.
2127 </para></listitem></varlistentry>
2128 <varlistentry><term><option>--session-dir=dir_path</option></term><listitem><para>
2129 Use sample database from the specified directory <filename>dir_path</filename> instead
2130 of the default location. If this option is not specified, then oparchive will search for
2131 samples in <filename><cur_dir>/oprofile_data</filename>
2132 first. If that directory does not exist, the standard session-dir of
2133 <filename>/var/lib/oprofile</filename> is used
2134 as the session directory.
2135 </para></listitem></varlistentry>
2136 <varlistentry><term><option>--version / -v</option></term><listitem><para>
2138 </para></listitem></varlistentry>
2141 </sect2> <!-- oparchive-details -->
2143 </sect1> <!-- oparchive -->
2145 <sect1 id="opimport">
2146 <title>Converting sample database files (<command>opimport</command>)</title>
2148 This utility converts sample database files from a foreign binary format (abi) to
2149 the native format. This is required when moving sample files to a (host) system
2150 other than the one used for collection (target system), and the host and target systems are different
2151 architectures. The abi format of the sample files to be imported is described in a
2152 text file located in <filename>$SESSION_DIR/abi</filename>. If you are unsure if
2153 your target and host systems have compatible architectures (in regard to the OProfile
2154 ABI), simply diff a <filename>$SESSION_DIR/abi</filename> file from the target system
2155 with one from the host system. If any differences show up at all, you must run the
2156 <command>opimport</command> command.
2160 The <command>oparchive</command> command should be used on the machine where
2161 the profile was taken (target) in order to collect sample files and all other necessary
2162 information. The archive directory that is the output from <command>oparchive</command>
2163 should be copied to the system where you wish to perform your performance analysis (host).
2167 The following command converts an input sample file to the specified
2168 output sample file using the given abi file as a binary description
2169 of the input file and the curent platform abi as a binary description
2170 of the output file. (NOTE: The ellipses are used to make the example more
2171 compact and cannot be used in an actual command line.)
2175 # opimport -a /tmp/foreign-abi -o /tmp/imported/.../GLOBAL_POWER_EVENTS.200000.1.all.all.all /tmp/archived/var/lib/.../mprime/GLOBAL_POWER_EVENTS.200000.1.all.all.all
2178 Since opimport converts just one file at a time, an example shell script is provided below
2179 that will perform an import/conversion of all sample files in a samples directory collected
2180 from the target system.
2183 #Usage: my-import.sh <foreign-abi-fullpathname>
2185 # NOTE: Start from the "samples" directory containing the "current" directory
2188 mkdir current-imported
2189 cd current-imported; (cd ../current; find . -type d ! -name .) |xargs mkdir
2190 cd ../current; mv stats ../StatsSave; find . -type f | while read line; do opimport -a $1 -o ../current-imported/$line $line; done; mv ../StatsSave stats;
2194 Example usage: Assume that on the target system, a profile was collected using a session-dir of
2195 <filename>/var/lib/oprofile</filename>, and then <command>oparchive -o profile1</command> was run.
2196 Then the <filename>profile1</filename> directory is copied to the host system for analysis. To import
2197 the sample data in <filename>profile1</filename>, you would perform the following steps:
2199 $cd profile1/var/lib/oprofile/samples
2200 $my-import.sh `pwd`/../abi
2204 If the OProfile ABI is truly different on host and target machines, then the end result of running the
2205 above script will place the converted (i.e., imported) files into the <filename>current-imported</filename>
2206 directory. By default, <command>opreport</command> and other post-profiling tools will look for samples
2207 in <filename>samples/current</filename> of the specified session directory. So you should either rename
2208 <filename>current-imported</filename> to <filename>current</filename> or specify the session specification of
2209 <command>session:current-imported</command> when running post-profiling tools.
2212 If the OProfile ABI is the same on the host and target machines, the <command>my-import.sh</command> script
2213 will print the following message for each sample file:
2215 input abi is identical to native. no conversion necessary.
2218 <sect2 id="opimport-details">
2219 <title>Usage of <command>opimport</command></title>
2222 <varlistentry><term><option>--help / -? / --usage</option></term><listitem><para>
2224 </para></listitem></varlistentry>
2225 <varlistentry><term><option>--abi / -a [filename]</option></term><listitem><para>
2226 Input abi file description location.
2227 </para></listitem></varlistentry>
2228 <varlistentry><term><option>--force / -f</option></term><listitem><para>
2229 Force conversion even if the input and output abi are identical.
2230 </para></listitem></varlistentry>
2231 <varlistentry><term><option>--output / -o [filename]</option></term><listitem><para>
2232 Specify the output filename. If the output file already exists, the file is
2233 not overwritten but data are accumulated in. Sample filename are informative
2234 for post profile tools and must be kept identical, in other word the pathname
2235 from the first path component containing a '{' must be kept as it in the
2237 </para></listitem></varlistentry>
2238 <varlistentry><term><option>--verbose / -V</option></term><listitem><para>
2239 Give verbose debugging output.
2240 </para></listitem></varlistentry>
2241 <varlistentry><term><option>--version / -v</option></term><listitem><para>
2243 </para></listitem></varlistentry>
2246 </sect2> <!-- opimport-details -->
2248 </sect1> <!-- opimport -->
2252 <chapter id="interpreting">
2253 <title>Interpreting profiling results</title>
2255 The standard caveats of profiling apply in interpreting the results from OProfile:
2256 profile realistic situations, profile different scenarios, profile
2257 for as long as a time as possible, avoid system-specific artifacts, don't trust
2258 the profile data too much. Also bear in mind the comments on the performance
2259 counters above - you <emphasis>cannot</emphasis> rely on totally accurate
2260 instruction-level profiling. However, for almost all circumstances the data
2261 can be useful. Ideally a utility such as Intel's VTUNE would be available to
2262 allow careful instruction-level analysis; go hassle Intel for this, not me ;)
2264 <sect1 id="irq-latency">
2265 <title>Profiling interrupt latency</title>
2267 This is an example of how the latency of delivery of profiling interrupts
2268 can impact the reliability of the profiling data. This is pretty much a
2269 worst-case-scenario example: these problems are fairly rare.
2272 double fun(double a, double b, double c)
2275 for (int i = 0 ; i < 10000; ++i) {
2284 Here the last instruction of the loop is very costly, and you would expect the result
2285 reflecting that - but (cutting the instructions inside the loop):
2288 $ opannotate -a -t 10 ./a.out
2290 88 15.38% : 8048337: fadd %st(3),%st
2291 48 8.391% : 8048339: fmul %st(2),%st
2292 68 11.88% : 804833b: fdiv %st(1),%st
2293 368 64.33% : 804833d: inc %eax
2294 : 804833e: cmp $0x270f,%eax
2295 : 8048343: jle 8048337
2298 The problem comes from the x86 hardware; when the counter overflows the IRQ
2299 is asserted but the hardware has features that can delay the NMI interrupt:
2300 x86 hardware is synchronous (i.e. cannot interrupt during an instruction);
2301 there is also a latency when the IRQ is asserted, and the multiple
2302 execution units and the out-of-order model of modern x86 CPUs also causes
2303 problems. This is the same function, with annotation :
2306 $ opannotate -s -t 10 ./a.out
2308 :double fun(double a, double b, double c)
2309 :{ /* _Z3funddd total: 572 100.0% */
2310 : double result = 0;
2311 368 64.33% : for (int i = 0 ; i < 10000; ++i) {
2312 88 15.38% : result += a;
2313 48 8.391% : result *= b;
2314 68 11.88% : result /= c;
2320 The conclusion: don't trust samples coming at the end of a loop,
2321 particularly if the last instruction generated by the compiler is costly. This
2322 case can also occur for branches. Always bear in mind that samples
2323 can be delayed by a few cycles from its real position. That's a hardware
2324 problem and OProfile can do nothing about it.
2327 <sect1 id="kernel-profiling">
2328 <title>Kernel profiling</title>
2329 <sect2 id="irq-masking">
2330 <title>Interrupt masking</title>
2332 OProfile uses non-maskable interrupts (NMI) on the P6 generation, Pentium 4,
2333 Athlon, Opteron, Phenom, and Turion processors. These interrupts can occur even in sections of the
2334 kernel where interrupts are disabled, allowing collection of samples in virtually
2335 all executable code.
2339 <title>Idle time</title>
2341 Your kernel is likely to support halting the processor when a CPU is idle. As
2342 the typical hardware events like <constant>CPU_CLK_UNHALTED</constant> do not
2343 count when the CPU is halted, the kernel profile will not reflect the actual
2344 amount of time spent idle. You can change this behaviour by booting with
2345 the <option>idle=poll</option> option, which uses a different idle routine. This
2346 will appear as <function>poll_idle()</function> in your kernel profile.
2349 <sect2 id="kernel-modules">
2350 <title>Profiling kernel modules</title>
2352 OProfile profiles kernel modules by default. However, there are a couple of problems
2353 you may have when trying to get results. First, you may have booted via an initrd;
2354 this means that the actual path for the module binaries cannot be determined automatically.
2355 To get around this, you can use the <option>-p</option> option to the analysis tools
2356 to specify where to look for the kernel modules.
2359 In kernel version 2.6, the information on where kernel module binaries are located was removed.
2360 This means OProfile needs guiding with the <option>-p</option> option to find your
2361 modules. Normally, you can just use your standard module top-level directory for this.
2362 Note that due to this problem, OProfile cannot check that the modification times match;
2363 it is your responsibility to make sure you do not modify a binary after a profile
2367 If you have run <command>insmod</command> or <command>modprobe</command> to insert a module
2368 in a particular directory, it is important that you specify this directory with the
2369 <option>-p</option> option first, so that it over-rides an older module binary that might
2370 exist in other directories you've specified with <option>-p</option>. It is up to you
2371 to make sure that these values are correct: the kernel simply does not provide enough
2372 information for OProfile to get this information.
2377 <sect1 id="interpreting-callgraph">
2378 <title>Interpreting call-graph profiles</title>
2380 Sometimes the results from call-graph profiles may be different from what
2381 you expect to see. The first thing to check is whether the target
2382 binaries where compiled with frame pointers enabled (if the binary was
2383 compiled using <command>gcc</command>'s
2384 <option>-fomit-frame-pointer</option> option, you will not get
2385 meaningful results). Note that as of this writing, the GCC developers
2386 plan to disable frame pointers by default. The Linux kernel is built
2387 without frame pointers by default; there is a configuration option you
2388 can use to turn it on under the "Kernel Hacking" menu.
2391 Often you may see a caller of a function that does not actually directly
2392 call the function you're looking at (e.g. if <function>a()</function>
2393 calls <function>b()</function>, which in turn calls
2394 <function>c()</function>, you may see an entry for
2395 <function>a()->c()</function>). What's actually occurring is that we
2396 are taking samples at the very start (or the very end) of
2397 <function>c()</function>; at these few instructions, we haven't yet
2398 created the new function's frame, so it appears as if
2399 <function>a()</function> is calling directly into
2400 <function>c()</function>. Be careful not to be misled by these
2404 Like the rest of OProfile, call-graph profiling uses a statistical
2405 approach; this means that sometimes a backtrace sample is truncated, or
2406 even partially wrong. Bear this in mind when examining results.
2408 <!-- FIXME: what do we need here ? -->
2411 <sect1 id="debug-info">
2412 <title>Inaccuracies in annotated source</title>
2413 <sect2 id="effect-of-optimizations">
2414 <title>Side effects of optimizations</title>
2416 The compiler can introduce some pitfalls in the annotated source output.
2417 The optimizer can move pieces of code in such manner that two line of codes
2418 are interlaced (instruction scheduling). Also debug info generated by the compiler
2419 can show strange behavior. This is especially true for complex expressions e.g. inside
2428 here the problem come from the position of line number. The available debug
2429 info does not give enough details for the if condition, so all samples are
2430 accumulated at the position of the right brace of the expression. Using
2431 <command>opannotate <option>-a</option></command> can help to show the real
2432 samples at an assembly level.
2435 <sect2 id="prologues">
2436 <title>Prologues and epilogues</title>
2438 The compiler generally needs to generate "glue" code across function calls, dependent
2439 on the particular function call conventions used. Additionally other things
2440 need to happen, like stack pointer adjustment for the local variables; this
2441 code is known as the function prologue. Similar code is needed at function return,
2442 and is known as the function epilogue. This will show up in annotations as
2443 samples at the very start and end of a function, where there is no apparent
2444 executable code in the source.
2447 <sect2 id="inlined-function">
2448 <title>Inlined functions</title>
2450 You may see that a function is credited with a certain number of samples, but
2451 the listing does not add up to the correct total. To pick a real example :
2454 :internal_sk_buff_alloc_security(struct sk_buff *skb)
2455 353 2.342% :{ /* internal_sk_buff_alloc_security total: 1882 12.48% */
2457 : sk_buff_security_t *sksec;
2458 15 0.0995% : int rc = 0;
2460 10 0.06633% : sksec = skb->lsm_security;
2461 468 3.104% : if (sksec && sksec->magic == DSI_MAGIC) {
2465 : sksec = (sk_buff_security_t *) get_sk_buff_memory(skb);
2466 3 0.0199% : if (!sksec) {
2467 38 0.2521% : rc = -ENOMEM;
2470 : memset(sksec, 0, sizeof (sk_buff_security_t));
2471 44 0.2919% : sksec->magic = DSI_MAGIC;
2472 32 0.2123% : sksec->skb = skb;
2473 45 0.2985% : sksec->sid = DSI_SID_NORMAL;
2474 31 0.2056% : skb->lsm_security = sksec;
2478 146 0.9685% : return rc;
2483 Here, the function is credited with 1,882 samples, but the annotations
2484 below do not account for this. This is usually because of inline functions -
2485 the compiler marks such code with debug entries for the inline function
2486 definition, and this is where <command>opannotate</command> annotates
2487 such samples. In the case above, <function>memset</function> is the most
2488 likely candidate for this problem. Examining the mixed source/assembly
2489 output can help identify such results.
2492 This problem is more visible when there is no source file available, in the
2493 following example it's trivially visible the sums of symbols samples is less
2494 than the number of the samples for this file. The difference must be accounted
2495 to inline functions.
2499 * Total samples for file : "arch/i386/kernel/process.c"
2504 /* default_idle total: 84 1.8970 */
2505 /* cpu_idle total: 21 0.4743 */
2506 /* flush_thread total: 1 0.0226 */
2507 /* prepare_to_copy total: 1 0.0226 */
2508 /* __switch_to total: 18 0.4065 */
2511 The missing samples are not lost, they will be credited to another source
2512 location where the inlined function is defined. The inlined function will be
2513 credited from multiple call site and merged in one place in the annotated
2514 source file so there is no way to see from what call site are coming the
2515 samples for an inlined function.
2518 When running <command>opannotate</command>, you may get a warning
2519 "some functions compiled without debug information may have incorrect source line attributions".
2520 In some rare cases, OProfile is not able to verify that the derived source line
2521 is correct (when some parts of the binary image are compiled without debugging
2522 information). Be wary of results if this warning appears.
2525 Furthermore, for some languages the compiler can implicitly generate functions,
2526 such as default copy constructors. Such functions are labelled by the compiler
2527 as having a line number of 0, which means the source annotation can be confusing.
2529 <!-- FIXME so what *actually* happens to those samples ? ignored ? -->
2531 <sect2 id="wrong-linenr-info">
2532 <title>Inaccuracy in line number information</title>
2534 Depending on your compiler you can fall into the following problem:
2537 struct big_object { int a[500]; };
2542 for (int i = 0 ; i != 1000 * 1000; ++i)
2549 Compiled with <command>gcc</command> 3.0.4 the annotated source is clearly inaccurate:
2553 :{ /* main total: 7871 100% */
2555 : for (int i = 0 ; i != 1000 * 1000; ++i)
2557 7871 100% : return 0;
2561 The problem here is distinct from the IRQ latency problem; the debug line number
2562 information is not precise enough; again, looking at output of <command>opannoatate -as</command> can help.
2568 : for (int i = 0 ; i != 1000 * 1000; ++i)
2569 : 80484c0: push %ebp
2570 : 80484c1: mov %esp,%ebp
2571 : 80484c3: sub $0xfac,%esp
2572 : 80484c9: push %edi
2573 : 80484ca: push %esi
2574 : 80484cb: push %ebx
2576 : 80484cc: lea 0xfffff060(%ebp),%edx
2577 : 80484d2: lea 0xfffff830(%ebp),%eax
2578 : 80484d8: mov $0xf423f,%ebx
2579 : 80484dd: lea 0x0(%esi),%esi
2581 3 0.03811% : 80484e0: mov %edx,%edi
2582 : 80484e2: mov %eax,%esi
2583 1 0.0127% : 80484e4: cld
2584 8 0.1016% : 80484e5: mov $0x1f4,%ecx
2585 7850 99.73% : 80484ea: repz movsl %ds:(%esi),%es:(%edi)
2586 9 0.1143% : 80484ec: dec %ebx
2587 : 80484ed: jns 80484e0
2588 : 80484ef: xor %eax,%eax
2596 So here it's clear that copying is correctly credited with of all the samples, but the
2597 line number information is misplaced. <command>objdump -dS</command> exposes the
2598 same problem. Note that maintaining accurate debug information for compilers when optimizing is difficult, so this problem is not suprising.
2599 The problem of debug information
2600 accuracy is also dependent on the binutils version used; some BFD library versions
2601 contain a work-around for known problems of <command>gcc</command>, some others do not. This is unfortunate but we must live with that,
2602 since profiling is pointless when you disable optimisation (which would give better debugging entries).
2606 <sect1 id="symbol-without-debug-info">
2607 <title>Assembly functions</title>
2609 Often the assembler cannot generate debug information automatically.
2610 This means that you cannot get a source report unless
2611 you manually define the neccessary debug information; read your assembler documentation for how you might
2613 debugging info needed currently by OProfile is the line-number/filename-VMA association. When profiling assembly
2614 without debugging info you can always get report for symbols, and optionally for VMA, through <command>opreport -l</command>
2615 or <command>opreport -d</command>, but this works only for symbols with the right attributes.
2616 For <command>gas</command> you can get this by
2623 whilst for <command>nasm</command> you must use
2626 GLOBAL foo:function ; [1]
2629 Note that OProfile does not need the global attribute, only the function attribute.
2634 FIXME: I commented this bit out until we've written something ...
2636 improve this ? but look first why this file is special
2637 <sect2 id="small-functions">
2638 <title>Small functions</title>
2640 Very small functions can show strange behavior. The file in your source
2641 directory of OProfile <filename>$SRC/test-oprofile/understanding/puzzle.c</filename>
2647 <sect1 id="overlapping-symbols">
2648 <title>Overlapping symbols in JITed code</title>
2650 Some virtual machines (e.g., Java) may re-JIT a method, resulting in previously
2651 allocated space for a piece of compiled code to be reused. This means that, at one distinct
2652 code address, multiple symbols/methods may be present during the run time of the application.
2655 Since OProfile samples are buffered and don′t have timing information, there is no way
2656 to correlate samples with the (possibly) varying address ranges in which the code for a symbol
2658 An alternative would be flushing the OProfile sampling buffer when we get an unload event,
2659 but this could result in high overhead.
2662 To moderate the problem of overlapping symbols, OProfile tries to select the symbol that was
2663 present at this address range most of the time. Additionally, other overlapping symbols
2664 are truncated in the overlapping area.
2665 This gives reasonable results, because in reality, address reuse typically takes place
2666 during phase changes of the application -- in particular, during application startup.
2667 Thus, for optimum profiling results, start the sampling session after application startup
2672 <sect1 id="interpreting_operf_results">
2673 <title>Using operf to profile fork/execs</title>
2675 When profiling an application that forks one or more new processes, <command>operf</command> will
2676 record samples for both the parent process and forked processes. This is also true even if the
2677 forked process performs an exec of some sort (e.g., <code>execvp</code>). If the
2678 process does <emphasis>not</emphasis> perform an exec, you will see that <command>opreport</command>
2679 will attribute samples for the forked process to the main application executable. On the other
2680 hand, if the forked process <emphasis>does</emphasis> perform an exec, then <command>opreport</command>
2681 will attribute samples to the executable being exec'ed.
2684 To demonstrate this, consider the following examples.
2685 When using <command>operf</command> to profile a single application (either with the <code>--pid</code>
2686 option or <code>command</code> option), the normal <command>opreport</command> summary output
2687 (i.e., invoking <command>opreport</command> with no options) looks something like the following:
2692 112342 100.000 sprintft
2696 104209 92.7605 libc-2.12.so
2697 7273 6.4740 sprintft
2698 858 0.7637 no-vmlinux
2703 But if you profile an application that does a fork/exec, the <command>opreport</command> summary output
2704 will show samples for both the main application you profiled, as well as the exec'ed program.
2705 An example is shown below where <code>s-m-fork</code> is the main application being profiled, which
2706 in turn forks a process that does an <code>execvp</code> of the <code>memcpyt</code> program.
2711 133382 70.5031 memcpyt
2715 123852 92.8551 libc-2.12.so
2717 1007 0.7550 no-vmlinux
2718 1 7.5e-04 ld-2.12.so
2719 55804 29.4969 s-m-fork
2723 51801 92.8267 libc-2.12.so
2724 3589 6.4314 s-m-fork
2725 414 0.7419 no-vmlinux
2730 <sect1 id="hidden-cost">
2731 <title>Other discrepancies</title>
2733 Another cause of apparent problems is the hidden cost of instructions. A very
2734 common example is two memory reads: one from L1 cache and the other from memory:
2735 the second memory read is likely to have more samples.
2736 There are many other causes of hidden cost of instructions. A non-exhaustive
2737 list: mis-predicted branch, TLB cache miss, partial register stall,
2738 partial register dependencies, memory mismatch stall, re-executed µops. If you want to write
2739 programs at the assembly level, be sure to take a look at the Intel and
2740 AMD documentation at <ulink url="http://developer.intel.com/">http://developer.intel.com/</ulink>
2741 and <ulink url="http://developer.amd.com/devguides.jsp/">http://developer.amd.com/devguides.jsp</ulink>.
2746 <chapter id="controlling-counter">
2747 <title>Controlling the event counter</title>
2748 <sect1 id="controlling-ocount">
2749 <title>Using <command>ocount</command></title>
2751 This section describes in detail how <command>ocount</command> is used.
2752 Unless the <option>--events</option> option is specified, <command>ocount</command> will use
2753 the default event for your system. For most systems, the default event is some
2754 cycles-based event, assuming your processor type supports hardware performance
2755 counters. The event specification used for <command>ocount</command> is slightly
2756 different from that required for profiling -- a <emphasis>count</emphasis> value
2757 is not needed. You can see the event information for your CPU using <command>ophelp</command>.
2758 More information on event specification can be found at <xref linkend="eventspec"/>.
2761 The <command>ocount</command> command syntax is:
2763 <screen>ocount [ options ] [ --system-wide | --process-list <pids> | --thread-list <tids> | --cpu-list <cpus> [ command [ args ] ] ]
2767 <command>ocount</command> has 5 run modes:
2770 <listitem>system-wide</listitem>
2771 <listitem>process-list</listitem>
2772 <listitem>thread-list</listitem>
2773 <listitem>cpu-list</listitem>
2774 <listitem>command</listitem>
2778 One and only one of these 5 run modes must be specified when you run <command>ocount</command>.
2779 If you run <command>ocount</command> using a run mode other than <code>command [args]</code>, press Ctrl-c
2780 to stop it when finished counting (e.g., when the monitored process ends). If you background <command>ocount</command>
2781 (i.e., with ’&’) while using one these run modes, you must stop it in a controlled manner so that
2782 the data collection process can be shut down cleanly and final results can be displayed.
2783 Use <code>kill -SIGINT <ocount-PID></code> for this purpose.
2786 Following is a description of the <command>ocount</command> options.
2790 <term><option>command [args]</option></term>
2792 The command or application to be profiled. The <emphasis>[args]</emphasis> are the input arguments
2793 that the command or application requires. The command and its arguments must be positioned at the
2794 end of the command line, after all other <command>ocount</command> options.
2798 <term><option>--process-list / -p [PIDs]</option></term>
2800 Use this option to count events for one or more already-running applications, specified via
2801 a comma-separated list (PIDs). Event counts will be collected for all children of the
2802 passed process(es) as well.
2806 <term><option>--thread-list / -r [TIDs]</option></term>
2808 Use this option to count events for one or more already-running threads, specified via
2809 a comma-separated list (TIDs). Event counts will <emphasis>not</emphasis> be collected
2810 for any children of the passed thread(s).
2814 <term><option>--system-wide / -s</option></term>
2816 This option is for counting events for all processes running on your system. You must have
2817 root authority to run <command>ocount</command> in this mode.
2821 <term><option>--cpu-list / -C [CPUs]</option></term>
2823 This option is for counting events on a subset of processors on your system. You must have
2824 root authority to run <command>ocount</command> in this mode. This is a comma-separated list,
2825 where each element in the list may be either a single processor number or a range of processor
2826 numbers; for example: ’-C 2,3,4-11,15’.
2830 <term><option>--events / -e [event1[,event2[,...]]]</option></term>
2832 This option is for passing a comma-separated list of event specifications
2833 for counting. Each event spec is of the form:
2835 <screen>name[:unitmask[:kernel[:user]]]</screen>
2837 When no event specification is given, the default event for the running
2838 processor type will be used for counting. Use <command>ophelp</command>
2839 to list the available events for your processor type.
2843 <term><option>--separate-thread / -t</option></term>
2845 This option can be used in conjunction with either the <code>--process-list</code> or
2846 <code>--thread-list</code> option to display event counts on a per-thread (per-process) basis.
2847 Without this option, all counts are aggregated.
2851 <term><option>--separate-cpu / -c</option></term>
2853 This option can be used in conjunction with either the <code>--system-wide</code> or
2854 <code>--cpu-list</code> option to display event counts on a per-cpu basis. Without this option,
2855 all counts are aggregated.
2859 <term><option>--time-interval / -i interval_length[:num_intervals]</option></term>
2861 <command>Note: </command>The <code>interval_length</code> is given in milliseconds.
2862 However, the current implementation only supports 100 ms
2863 granularity, so the given <code>interval_length</code> will be rounded
2864 to the nearest 100 ms. Results collected for each time
2865 interval are printed immediately instead of the default
2866 of one dump of cumulative event counts at the end of the
2867 run. Counters are reset to zero at the start of each
2871 If <code>num_intervals</code> is specified, ocount exits after the
2872 specified number of intervals occur.
2876 <term><option>--brief-format / -b</option></term>
2878 Use this option to print results in the following brief format:
2880 [optional cpu or thread,]<event_name>,<count>,<percent_time_enabled>
2881 [ <int> ,]< string >,< u64 >,< double >
2883 If <code>--timer-interval</code> is specified, a separate line formatted as
2885 timestamp,<num_seconds_since_epoch>[.n]
2887 is printed ahead of each dump of event counts. If the time interval specified is
2888 less than one second, the timestamp will have 1/10 second precision.
2893 <term><option>--output-file / -f outfile_name</option></term>
2895 Results are written to outfile_name instead of interactively to the terminal.
2899 <term><option>--verbose / -V</option></term>
2901 Use this option to increase the verbosity of the output.
2905 <term><option>--version -v </option></term>
2907 Show <command>ocount</command> version.
2911 <term><option>--help / -h</option></term>
2913 Show a help message.
2923 <title>Acknowledgments</title>
2925 Thanks to (in no particular order) : Arjan van de Ven, Rik van Riel, Juan Quintela, Philippe Elie,
2926 Phillipp Rumpf, Tigran Aivazian, Alex Brown, Alisdair Rawsthorne, Bob Montgomery, Ray Bryant, H.J. Lu,
2927 Jeff Esper, Will Cohen, Graydon Hoare, Cliff Woolley, Alex Tsariounov, Al Stone, Jason Yeh,
2928 Randolph Chung, Anton Blanchard, Richard Henderson, Andries Brouwer, Bryan Rittmeyer,
2930 Richard Reich (rreich@rdrtech.com), Zwane Mwaikambo, Dave Jones, Charles Filtness; and finally Pulp, for "Intro".
projects / platform / upstream / oprofile.git / blob