1 <!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.1.2//EN" "http://www.oasis-open.org/docbook/xml/4.1.2/docbookx.dtd">
3 <book id="oprofile-guide">
5 <title>OProfile manual</title>
9 <firstname>John</firstname>
10 <surname>Levon</surname>
12 <address><email>levon@movementarian.org</email></address>
18 <year>2000-2004</year>
19 <holder>Victoria University of Manchester, John Levon and others</holder>
25 <chapter id="introduction">
26 <title>Introduction</title>
29 This manual applies to OProfile version <oprofileversion />.
30 OProfile is a set of performance monitoring tools for Linux 2.6 and higher systems, available on a number of architectures.
31 OProfile provides the following features:
33 <listitem>Profiler</listitem>
34 <listitem>Post-processing tools for analyzing profile data</listitem>
35 <listitem>Event counter</listitem>
39 OProfile is capable of monitoring native hardware events occurring in all parts of a running system, from the kernel
40 (including modules and interrupt handlers) to shared libraries
41 to binaries. OProfile can collect event information for the whole system in the background with very little overhead. These
42 features make it ideal for monitoring entire systems to determine bottle necks in real-world systems.
46 Many CPUs provide "performance counters", hardware registers that can count "events"; for example,
47 cache misses, or CPU cycles. OProfile can collect profiles of code based on the number of these occurring events:
48 repeatedly, every time a certain (configurable) number of events has occurred, the PC value is recorded.
49 This information is aggregated into profiles for each binary image. Alternatively, OProfile's event counting
50 tool can collect simple raw event counts.</para>
51 <sect1 id="legacy_mode">
52 <title>OProfile legacy profiling mode</title>
53 Prior to release 1.0, OProfile included a profiling tool consisting of the <command>opcontrol</command> shell script, the <command>oprofiled</command> daemon,
54 and the attendant oprofile kernel driver. This "legacy profiler" was deprecated in release 0.9.8 with the introduction of
55 the <command>operf</command> profiling tool (see <xref linkend="perf_events"/>). Some older architectures/platforms
56 do not support the use of <command>operf</command>. For those cases, oprofile users should install release 0.9.9, which is the
57 last release to include the legacy profiler.
59 <sect1 id="perf_events">
60 <title>OProfile perf_events profiling mode</title>
62 OProfile has the ability to profile a single process or every currently running process (i.e., system-wide)
63 via the <command>operf</command> program. <command>operf</command> interfaces with the
64 kernel to collect samples via the Linux Kernel Performance Events Subsystem (hereafter
65 referred to as "perf_events"). OProfile can co-exist with other tools on your system that
66 may also be using the perf_events kernel subsystem.
69 Using <command>operf</command> to profile a single
70 process can be done as a normal user; however, root authority <emphasis>is</emphasis> required to run
71 <command>operf</command> in system-wide profiling mode.
74 Some older processor models are not supported by the underlying perf_events kernel and, thus, are not supported by <command>operf</command>.
75 If you receive the message
76 <screen> Your kernel's Performance Events Subsystem does not support your processor type</screen>
77 when attempting to use <command>operf</command>, install OProfile 0.9.9 and try profiling with <command>opcontrol</command>
78 to see if your processor type may be supported by OProfile's legacy mode.
83 <sect1 id="event_counting">
84 <title>OProfile event counting mode</title>
85 OProfile provides the <command>ocount</command> tool for
86 collecting raw event counts on a per-application, per-process, per-cpu, or system-wide basis. Unlike the
87 profiling tools, post-processing of the data collected is not necessary -- the data is displayed in the
88 output of <command>ocount</command>. A common use case for event counting tools is when performance analysts
89 want to determine the CPI (cycles per instruction) for an application. High CPI implies possible stalls,
90 and many architectures provide events that give detailed information about the different types of stalls.
91 The events provided are architecture-specific, so we refer the reader to the hardware manuals available for
92 the processor type being used.
96 <sect1 id="applications">
97 <title>Applications of OProfile</title>
99 OProfile is useful in a number of situations. You might want to use OProfile when you :
102 <listitem><para>need low overhead</para></listitem>
103 <listitem><para>cannot use highly intrusive profiling methods</para></listitem>
104 <listitem><para>need to profile interrupt handlers</para></listitem>
105 <listitem><para>need to profile an application and its shared libraries</para></listitem>
106 <listitem><para>need to profile dynamically compiled code of supported virtual machines (see <xref linkend="jitsupport"/>)</para></listitem>
107 <listitem><para>need to capture the performance behaviour of entire system</para></listitem>
108 <listitem><para>want to examine hardware effects such as cache misses</para></listitem>
109 <listitem><para>want detailed source annotation</para></listitem>
110 <listitem><para>want instruction-level profiles</para></listitem>
111 <listitem><para>want call-graph profiles</para></listitem>
114 OProfile is not a panacea. OProfile might not be a complete solution when you :
117 <listitem><para>require call graph profiles on platforms other than x86, ARM, and PowerPC</para></listitem>
118 <listitem><para>require 100% instruction-accurate profiles</para></listitem>
119 <listitem><para>need function call counts or an interstitial profiling API</para></listitem>
120 <listitem><para>cannot tolerate any disturbance to the system whatsoever</para></listitem>
121 <listitem><para>need to profile interpreted or dynamically compiled code of non-supported virtual machines</para></listitem>
123 <sect2 id="jitsupport">
124 <title>Support for dynamically compiled (JIT) code</title>
126 OProfile provides a framework to support JITed code ("just-in-time (JIT) compiled code").
127 A development library is provided to allow developers
128 to add support for any VM (virtual machine) that produces dynamically compiled code (see the <emphasis>OProfile JIT agent
129 developer guide</emphasis>).
130 In addition, built-in support is included for the following:</para>
131 <itemizedlist><listitem>JVMTI agent library for Java (1.5 and higher)</listitem>
132 <listitem>JVMPI agent library for Java (1.5 and lower)</listitem>
134 These libraries make it possible for OProfile to attribute profile samples
135 to Java methods. Without a VM-specific agent library, OProfile will typically report
136 samples from JITed code similar to the following example:
137 <screen> anon: <tgid><address range></screen>
138 For information on how to use OProfile's JIT support, see <xref linkend="setup-jit"/>.
141 <sect2 id="guestsupport">
142 <title>No support for virtual machine guests</title>
144 OProfile currently does not support event-based profiling (i.e, using hardware events like cache misses,
145 branch mispredicts) on virtual machine guests running under systems such as VMware.
146 (Note: KVM guests <emphasis>are</emphasis> supported.) The list of
147 supported events displayed by ophelp is based on CPU type and does
148 not take into account whether the running system is a guest system or real system. To use
149 OProfile on such guest systems, you must use the legacy profiler's timer mode (see <xref linkend="timer" />).
156 <sect1 id="requirements">
157 <title>System requirements</title>
161 <term>Linux kernel</term>
163 Release 2.6.31 or higher
167 <term>Supported architectures</term>
169 AMD, ARM, Intel, PowerPC, Tile, MIPS
173 <term>Required libraries</term>
175 These libraries are required : <filename>popt</filename>, <filename>bfd</filename>,
176 <filename>liberty</filename> (debian users: libiberty is provided in binutils-dev package), <filename>dl</filename>,
177 plus the standard C++ libraries.
181 <term>Required kernel headers</term>
183 Either the kernel-headers package must be installed or use the <code>--with-kernel</code>
188 <term>Required user account</term>
190 For secure processing of sample data from JIT virtual machines (e.g., Java),
191 the special user account "oprofile" must exist on the system. The 'configure'
192 and 'make install' operations will print warning messages if this
193 account is not found. If you intend to profile JITed code, you must create
194 a group account named 'oprofile' and then create the 'oprofile' user account,
195 setting the default group to 'oprofile'. A runtime error message is printed to
196 the oprofile log when processing JIT samples if this special user
197 account cannot be found.
201 <term><acronym>ELF</acronym></term>
203 Probably not too strenuous a requirement, but older <acronym>A.OUT</acronym> binaries/libraries are not supported.
207 <term>K&R coding style</term>
209 OK, so it's not really a requirement, but I wish it was...
217 <sect1 id="resources">
218 <title>Internet resources</title>
222 <term>Web page</term>
224 There is a web page (which you may be reading now) at
225 <ulink url="http://oprofile.sf.net/">http://oprofile.sf.net/</ulink>.
229 <term>Download</term>
231 You can download a source tarball or check out code from
232 the code repository at the sourceforge page,
233 <ulink url="http://sf.net/projects/oprofile/">http://sf.net/projects/oprofile/</ulink>.
237 <term>Mailing list</term>
239 There is a low-traffic OProfile-specific mailing list, details at
240 <ulink url="http://sf.net/mail/?group_id=16191">http://sf.net/mail/?group_id=16191</ulink>.
244 <term>Bug tracker</term>
246 There is a bug tracker for OProfile at SourceForge,
247 <ulink url="http://sourceforge.net/p/oprofile/bugs/">http://sourceforge.net/p/oprofile/bugs/</ulink>.
251 <term>IRC channel</term>
253 Several OProfile developers and users sometimes hang out on channel <command>#oprofile</command>
254 on the <ulink url="http://oftc.net">OFTC</ulink> network.
262 <title>Installation</title>
265 First you need to build OProfile and install it. <command>./configure</command>, <command>make</command>, <command>make install</command>
266 is often all you need, but note these arguments to <command>./configure</command> :
270 <term><option>--with-java</option></term>
273 Use this option if you need to profile Java applications. Also, see
274 <xref linkend="requirements"/>, "Required user account". This option
275 is used to specify the location of the Java Development Kit (JDK)
276 source tree you wish to use. This is necessary to get the interface description
277 of the JVMPI (or JVMTI) interface to compile the JIT support code successfully.
281 The Java Runtime Environment (JRE) does not include the development
282 files that are required to compile the JIT support code, so the full
283 JDK must be installed in order to use this option.
287 By default, the Oprofile JIT support libraries will be installed in
288 <filename><oprof_install_dir>/lib/oprofile</filename>. To build
289 and install OProfile and the JIT support libraries as 64-bit, you can
290 do something like the following:
292 # CFLAGS="-m64" CXXFLAGS="-m64" ./configure \
293 --with-java={my_jdk_installdir} \
294 --libdir=/usr/local/lib64
299 If you encounter errors building 64-bit, you should
300 install libtool 1.5.26 or later since that release of
301 libtool fixes known problems for certain platforms.
302 If you install libtool into a non-standard location,
303 you'll need to edit the invocation of 'aclocal' in
304 OProfile's autogen.sh as follows (assume an install
305 location of /usr/local):
308 <code>aclocal -I m4 -I /usr/local/share/aclocal</code>
314 <term><option>--with-qt-dir/includes/libraries</option></term>
316 Specify the location of Qt headers and libraries. It defaults to searching in
317 <constant>$QTDIR</constant> if these are not specified.
320 <varlistentry id="disable-werror">
321 <term><option>--disable-werror</option></term>
323 Development versions of OProfile build by
324 default with <option>-Werror</option>. This option turns
325 <option>-Werror</option> off.
328 <varlistentry id="disable-optimization">
329 <term><option>--disable-optimization</option></term>
331 Disable the <option>-O2</option> compiler flag
332 (useful if you discover an OProfile bug and want to give a useful
336 <varlistentry id="with-kernel">
337 <term><option>--with-kernel</option></term>
339 This option is used to specify the location of the kernel headers <filename>include</filename> directory
340 needed to build the perf_events-enabled <command>operf</command> program. By default, the OProfile
341 build system expects to find this directory under <filename>/usr</filename>. Use this option if your
342 kernel headers are in a non-standard location or if building in a cross-compile enviroment or in a
343 situation where the host system does not support perf_events but you wish to build binaries for a
344 target system that does support perf_events.
349 It is recommended that if you have a
350 uniprocessor machine, you enable the local APIC / IO_APIC support for
351 your kernel (this is automatically enabled for SMP kernels). With many BIOS (kernel >= 2.6.9 and UP kernel)
352 it's not sufficient to enable the local APIC -- you must also turn it on explicitly at boot
353 time by providing the "lapic" option to the kernel.
354 If you use the NMI watchdog, be aware that the watchdog is disabled when profiling starts
355 and not re-enabled until the profiling is stopped.
360 <sect1 id="uninstall">
361 <title>Uninstalling OProfile</title>
363 You must have the source tree available to uninstall OProfile; a <command>make uninstall</command> will
364 remove all installed files except your configuration file in the directory <filename>~/.oprofile</filename>.
370 <chapter id="overview">
371 <title>Overview</title>
372 <sect1 id="getting-started-with-operf">
373 <title>Getting started with OProfile using <command>operf</command></title>
375 Profiling with <command>operf</command> allows you to precisely target your profiling (i.e., single process
376 or system-wide). With <command>operf</command>, there is no initial setup needed -- simply invoke <command>operf</command> with
377 the options you need; then run the OProfile post-processing tool(s). The <command>operf</command> syntax
380 <screen>operf [ options ] [ --system-wide | --pid=<PID> | [ command [ args ] ] ]</screen>
382 A typical usage might look like this:
384 <screen>operf ./my_test_program my_arg</screen>
386 When <filename>./my_test_program</filename> completes (or when you press Ctrl-C), profiling
387 stops and you're ready to use <command>opreport</command> or other OProfile post-processing tools.
388 By default, <command>operf</command> stores the sample data in <filename><cur_dir>/oprofile_data/samples/current</filename>,
389 and <command>opreport</command> and other post-processing tools will look in that location first for profile data,
390 unless you pass the <code>--session-dir</code> option.
395 <sect1 id="getting-started-with-ocount">
396 <title>Getting started with OProfile using <command>ocount</command></title>
398 <command>ocount</command> is an OProfile tool that can be used to count native hardware events occurring in either
399 a specific application, a set of processes or threads, a set of active system processors, or the
400 entire system. The data collected during a counting session is displayed to stdout by default, but may
401 also be saved to a file. The <command>ocount</command> syntax is as follows:
403 <screen>ocount [ options ] [ --system-wide | --process-list <pids> | --thread-list <tids> | --cpu-list <cpus> [ command [ args ] ] ]
406 A typical usage might look like this:
408 <screen>ocount --events=CPU_CLK_UNHALTED,INST_RETIRED /home/user1/my_test_program my_arg</screen>
410 When <filename>my_test_program</filename> completes (or when you press Ctrl-C), counting
411 stops and the results are displayed to the screen (as shown below).
414 Events were actively counted for 2.8 seconds.
415 Event counts (actual) for /home/user1/my_test_program:
416 Event Count % time counted
417 CPU_CLK_UNHALTED 9,408,018,070 100.00
418 INST_RETIRED 16,719,918,108 100.00
424 <sect1 id="eventspec">
425 <title>Specifying performance counter events</title>
427 Whether profiling with <command>operf</command> or doing simple event counting with <command>ocount</command>,
428 you can collect information about one more native hardware events using the <code>--events</code>
429 option -- a comma-separated list of event specfications. The event specification is the means to provide details
430 of how each hardware performance counter should be set up.
431 For profiling, the event specification is a colon-separated string of the form
432 <option><emphasis>name</emphasis>:<emphasis>count</emphasis>:<emphasis>unitmask</emphasis>:<emphasis>kernel</emphasis>:<emphasis>user</emphasis></option>
433 as described in the table below. For <command>ocount</command>, specification is of the form
434 <option><emphasis>name</emphasis>:<emphasis>unitmask</emphasis>:<emphasis>kernel</emphasis>:<emphasis>user</emphasis></option>.
435 Note the presence of the <emphasis>count</emphasis> field for profiling. The <emphasis>count</emphasis> field tells the profiler
436 how many events should occur between a profile snapshot (usually referred to as a "sample"). Since
437 <command>ocount</command> does not do sampling, the <emphasis>count</emphasis> field is not needed.
440 If no event specs are passed to <command>operf</command> or <command>ocount</command>,
441 the default event will be used.
444 <note>The perf_events kernel subsystem allocates hardware counters as necessary, but some processor
445 types have restrictions as to what hardware events may be counted simultaneously.
446 The kernel employs a multiplexing technique when such
447 hardware restrictions are encountered, such that events are monitored on a rotating basis.
450 <informaltable frame="all">
453 <row><entry><option>name</option></entry><entry>The symbolic event name, e.g. <constant>CPU_CLK_UNHALTED</constant></entry></row>
454 <row><entry><option>count</option></entry><entry>The counter reset value, e.g. 100000; use only for profiling</entry></row>
455 <row><entry><option>unitmask</option></entry><entry>The unit mask, as given in the events list: e.g. 0x0f; or a symbolic name
456 if a <constant>name=<um_name></constant> field is present</entry></row>
457 <row><entry><option>kernel</option></entry><entry>Enable profiling of kernel code</entry></row>
458 <row><entry><option>user</option></entry><entry>Enable profiling of userspace code</entry></row>
463 The last three values are optional; if you omit them (e.g. <option>operf --events=DATA_MEM_REFS:30000</option>),
464 they will be set to the default values (i.e., the default unit mask value for the given event, and profiling (or counting)
465 both kernel and userspace code will be enabled). Note that on some architectures, some events may
466 require a unit mask be specified.
469 You can specify unit mask values using either a numerical value (hex values
470 <emphasis>must</emphasis> begin with "0x") or a symbolic name (if the <constant>name=<um_name></constant>
471 field is shown in the <command>ophelp</command> output). For some named unit masks, the hex value is not unique; thus, OProfile
472 tools enforce specifying such unit masks value by name.
475 The table below lists the default profiling event for various processor types. The same events
476 can be used for <command>ocount</command>, minus the <emphasis>count</emphasis> field.
478 <informaltable frame="all">
481 <row><entry>Processor</entry><entry>cpu_type</entry><entry>Default event</entry></row>
482 <row><entry>Alpha EV67</entry><entry>alpha/ev67</entry><entry>CYCLES:100000:0:1:1</entry></row>
483 <row><entry>ARM/XScale PMU1</entry><entry>arm/xscale1</entry><entry>CPU_CYCLES:100000:0:1:1</entry></row>
484 <row><entry>ARM/XScale PMU2</entry><entry>arm/xscale2</entry><entry>CPU_CYCLES:100000:0:1:1</entry></row>
485 <row><entry>ARM/MPCore</entry><entry>arm/mpcore</entry><entry>CPU_CYCLES:100000:0:1:1</entry></row>
486 <row><entry>Athlon</entry><entry>i386/athlon</entry><entry>CPU_CLK_UNHALTED:100000:0:1:1</entry></row>
487 <row><entry>Pentium Pro</entry><entry>i386/ppro</entry><entry>CPU_CLK_UNHALTED:100000:0:1:1</entry></row>
488 <row><entry>Pentium II</entry><entry>i386/pii</entry><entry>CPU_CLK_UNHALTED:100000:0:1:1</entry></row>
489 <row><entry>Pentium III</entry><entry>i386/piii</entry><entry>CPU_CLK_UNHALTED:100000:0:1:1</entry></row>
490 <row><entry>Pentium M (P6 core)</entry><entry>i386/p6_mobile</entry><entry>CPU_CLK_UNHALTED:100000:0:1:1</entry></row>
491 <row><entry>Pentium 4 (non-HT)</entry><entry>i386/p4</entry><entry>GLOBAL_POWER_EVENTS:100000:1:1:1</entry></row>
492 <row><entry>Pentium 4 (HT)</entry><entry>i386/p4-ht</entry><entry>GLOBAL_POWER_EVENTS:100000:1:1:1</entry></row>
493 <row><entry>Hammer</entry><entry>x86-64/hammer</entry><entry>CPU_CLK_UNHALTED:100000:0:1:1</entry></row>
494 <row><entry>Family10h</entry><entry>x86-64/family10</entry><entry>CPU_CLK_UNHALTED:100000:0:1:1</entry></row>
495 <row><entry>Family11h</entry><entry>x86-64/family11h</entry><entry>CPU_CLK_UNHALTED:100000:0:1:1</entry></row>
496 <row><entry>IBM pseries</entry><entry>ppc64/power{ 4|5|6|7|8|970 }</entry><entry>CYCLES:100000:0:1:1</entry></row>
497 <row><entry>IBM s390</entry><entry>s390/{ z10|z196|zEC12 }</entry><entry>HWSAMPLING:4127518:0:1:1</entry></row>
504 <sect1 id="tools-overview">
505 <title>Tools summary</title>
507 This section gives a brief description of the available OProfile utilities and their purpose.
511 <term><filename>ophelp</filename></term>
513 This utility lists the available events and short descriptions.
518 <term><filename>operf</filename></term>
520 This is the program for collecting profile data, discussed in <xref linkend="controlling-operf" />.
525 <term><filename>ocount</filename></term>
527 This tool is used for simple event counting, as described in in <xref linkend="controlling-ocount" />.
532 <term><filename>agent libraries</filename></term>
534 Used by virtual machines (like the Java VM) to record information about JITed code being profiled. See <xref linkend="setup-jit" />.
539 <term><filename>opreport</filename></term>
541 This is the main tool for retrieving useful profile data, described in
542 <xref linkend="opreport" />.
547 <term><filename>opannotate</filename></term>
549 This utility can be used to produce annotated source, assembly or mixed source/assembly.
550 Source level annotation is available only if the application was compiled with
551 debugging symbols. See <xref linkend="opannotate" />.
556 <term><filename>opgprof</filename></term>
558 This utility can output gprof-style data files for a binary, for use with
559 <command>gprof -p</command>. See <xref linkend="opgprof" />.
564 <term><filename>oparchive</filename></term>
566 This utility can be used to collect executables, debuginfo,
567 and sample files and copy the files into an archive.
568 The archive is self-contained and can be moved to another
569 machine for further analysis.
570 See <xref linkend="oparchive" />.
575 <term><filename>opimport</filename></term>
577 This utility converts sample database files from a foreign binary format (abi) to
578 the native format. This is useful only when moving sample files between hosts
579 for analysis on platforms other than the one used for collection.
580 See <xref linkend="opimport" />.
589 <chapter id="controlling-profiler">
590 <title>Controlling the profiler</title>
592 <sect1 id="controlling-operf">
593 <title>Using <command>operf</command></title>
595 This section describes in detail how <command>operf</command> is used to
596 control profiling. Unless otherwise directed, <command>operf</command> will profile using
597 the default event for your system. For most systems, the default event is some
598 cycles-based event, assuming your processor type supports hardware performance
599 counters. If your hardware <emphasis>does</emphasis> support performance counters, you can specify
600 something other than the default hardware event on which to profile. The performance
601 monitor counters can be programmed to count various hardware events,
602 such as cache misses or MMX operations. The event
603 chosen for each counter is reflected in the profile data collected
604 by OProfile: functions and binaries at the top of the profiles reflect
605 that most of the chosen events happened within that code.
608 Additionally, each counter is programmed with a "count" value, which corresponds to how
609 detailed the profile is. The lower the value, the more frequently profile
610 samples are taken. You can choose to sample only kernel code, user-space code,
611 or both (both is the default). Finally, some events have a "unit mask"
612 -- this is a value that further restricts the type of event being counted.
613 You can see the event types and unit masks for your CPU using <command>ophelp</command>.
614 More information on event specification can be found at <xref linkend="eventspec"/>.
617 The <command>operf</command> command syntax is:
619 <screen>operf [ options ] [ --system-wide | --pid=<PID> | [ command [ args ] ] ]</screen>
621 When profiling an application using either the <code>command</code> or <code>--pid</code> option of
622 <command>operf</command>, forks and execs of the profiled process will also be profiled. The samples
623 from an exec'ed process will be attributed to the executable binary run by that process. See
624 <xref linkend="interpreting_operf_results"/>
627 Following is a description of the <command>operf</command> options.
631 <term><option>command [args]</option></term>
633 The command or application to be profiled. The <emphasis>[args]</emphasis> are the input arguments
634 that the command or application requires. Either <code>command</code>, <code>--pid</code> or
635 <code>--system-wide</code> is required, but cannot be used simultaneously.
639 <term><option>--pid / -p [PID]</option></term>
641 This option enables <command>operf</command> to profile a running application. <code>PID</code>
642 should be the process ID of the process you wish to profile. When
643 finished profiling (e.g., when the profiled process ends), press
644 Ctrl-c to stop <command>operf</command>.
648 <term><option>--system-wide / -s</option></term>
650 This option is for performing a system-wide profile. You must
651 have root authority to run <command>operf</command> in this mode.
652 When finished profiling, Ctrl-C to stop <command>operf</command>. If you run
653 <code>operf --system-wide</code> as a background job (i.e., with the &), you
654 <emphasis>must</emphasis> stop it in a controlled manner in order to process
655 the profile data it has collected. Use <code>kill -SIGINT <operf-PID></code>
656 for this purpose. It is recommended that when running <command>operf</command>
657 with this option, your current working directory should be <filename>/root</filename> or a subdirectory
658 of <filename>/root</filename> to avoid storing sample data files in locations accessible by regular users.
662 <term><option>--vmlinux / k [vmlinux_path]</option></term>
664 A vmlinux file that matches the running kernel that has symbol and/or debuginfo.
665 Kernel samples will be attributed to this binary, allowing post-processing tools
666 (like <command>opreport</command>) to attribute samples to the appropriate kernel symbols.
667 If this option is not specified, the file /proc/kallsyms is used to obtain
668 kernel symbol addresses correponding to sample addresses. However, the setting of
669 /proc/sys/kernel/kptr_restrict may restrict a non-root user's access to
670 /proc/kallsyms, in which case,
671 all kernel samples are attributed to a pseudo binary named "no-vmlinux".
675 <term><option>--callgraph / -g</option></term>
677 This option enables the callgraph to be saved during profiling. NOTE: The
678 full callchain is recorded, so there is no depth limit.
682 <term><option>--append / -a</option></term>
684 By default, <command>operf</command> moves old profile data from
685 <filename><session_dir>/samples/current</filename> to
686 <filename><session_dir>/samples/previous</filename>.
687 If a 'previous' profile already existed, it will be replaced. If the
688 <code>--append</code> option is passed, old profile data in 'current' is left in place and
689 new profile data will be added to it, and the 'previous' profile (if one existed)
690 will remain untouched. To access the 'previous' profile, simply add a session
691 specification to the normal invocation of oprofile post-processing tools; for example:
694 <screen>opreport session:previous</screen>
698 <term><option>--events / -e [event1[,event2[,...]]]</option></term>
700 This option is for passing a comma-separated list of event specifications
701 for profiling. Each event spec is of the form:
703 <screen>name:count[:unitmask[:kernel[:user]]]</screen>
705 When no event specification is given, the default event for the running
706 processor type will be used for profiling. Use <command>ophelp</command>
707 to list the available events for your processor type.
711 <term><option>--separate-thread / -t</option></term>
713 This option categorizes samples by thread group ID (tgid) and thread ID (tid).
714 The <code>--separate-thread</code> option is useful for seeing per-thread samples in
715 multi-threaded applications. When used in conjuction with the <code>--system-wide</code>
716 option, <code>--separate-thread</code> is also useful for seeing per-process
717 (i.e., per-thread group) samples for the case where multiple processes are
718 executing the same program during a profiling run.
722 <term><option>--separate-cpu / -c</option></term>
724 This option categorizes samples by cpu.
728 <term><option>--session-dir / -d [path]</option></term>
730 This option specifies the session directory to hold the sample data. If not specified,
731 the data is saved in the <filename>oprofile_data</filename> directory on the current path.
735 <term><option>---lazy-conversion / -l</option></term>
737 Use this option to reduce the overhead of <command>operf</command> during profiling.
738 Normally, profile data received from the kernel is converted to OProfile format
739 during profiling time. This is typically not an issue when profiling a single
740 application. But when using the <code>--system-wide</code> option, this on-the-fly
741 conversion process can cause noticeable overhead, particularly on busy
742 multi-processor systems. The <code>--lazy-conversion</code> option directs
743 <command>operf</command> to wait until profiling is completed to do the conversion
748 <term><option>--verbose / -V [level]</option></term>
750 A comma-separated list of debugging control values used to increase the verbosity of the
751 output. Valid values are: debug, record, convert, misc, sfile, arcs, and the special value, 'all'.
755 <term><option>--version -v </option></term>
757 Show <command>operf</command> version.
761 <term><option>--help / -h</option></term>
770 <sect1 id="setup-jit">
771 <title>Setting up the JIT profiling feature</title>
773 To gather information about JITed code from a virtual machine,
774 it needs to be instrumented with an agent library. We use the
775 agent libraries for Java in the following example. To use the
776 Java profiling feature, you must build OProfile with the "--with-java" option
777 (<xref linkend="install" />).
781 <sect2 id="setup-jit-jvm">
782 <title>JVM instrumentation</title>
784 Add this to the startup parameters of the JVM (for JVMTI):
786 <screen><option>-agentpath:<libdir>/libjvmti_oprofile.so[=<options>]</option> </screen>
788 <screen><option>-agentlib:jvmti_oprofile[=<options>]</option> </screen>
791 The JVMPI agent implementation is enabled with the command line option
792 <screen><option>-Xrunjvmpi_oprofile[:<options>]</option> </screen>
795 Currently, there is just one option available -- <option>debug</option>. For JVMPI,
796 the convention for specifying an option is <option>option_name=[yes|no]</option>.
797 For JVMTI, the option specification is simply the option name, implying
798 "yes"; no option specified implies "no".
801 The agent library (installed in <filename><oprof_install_dir>/lib/oprofile</filename>)
802 needs to be in the library search path (e.g. add the library directory
803 to <constant>LD_LIBRARY_PATH</constant>). If the command line of
804 the JVM is not accessible, it may be buried within shell scripts or a
805 launcher program. It may also be possible to set an environment variable to add
807 For Sun JVMs this is <constant>JAVA_TOOL_OPTIONS</constant>. Please check
808 your JVM documentation for
809 further information on the agent startup options.
816 <sect1 id="detailed-parameters">
817 <title>Configuration details</title>
819 <sect2 id="hardware-counters">
820 <title>Hardware performance counters</title>
821 <para>Most processor models include performance monitor units that can be configured to monitor (count)
822 various types of hardware events. This section is where you can find architecture-specific information
823 to help you use these events for profiling. You do not really need to read this section unless you are interested in using
824 events other than the default event chosen by OProfile.
828 Your CPU type may not include the requisite support for hardware performance counters, in which case
829 you must use OProfile in timer mode (see <xref linkend="timer" />), which is only available in
830 OProfile releases prior to 1.0.
834 The Intel hardware performance counters are detailed in the Intel IA-32 Architecture Manual, Volume 3, available
835 from <ulink url="http://developer.intel.com/">http://developer.intel.com/</ulink>.
836 The AMD Athlon/Opteron/Phenom/Turion implementation is detailed in <ulink
837 url="http://www.amd.com/us-en/assets/content_type/white_papers_and_tech_docs/22007.pdf">
838 http://www.amd.com/us-en/assets/content_type/white_papers_and_tech_docs/22007.pdf</ulink>.
839 For IBM PowerPC processors, documentation is available at <ulink url="https://www.power.org/">
840 https://www.power.org/</ulink>. For example, <ulink url="https://www.power.org/events/Power7">
841 https://www.power.org/events/Power7</ulink> contains specific information on the performance
842 monitor unit for the IBM POWER7.
845 A physical performance monitor counter (PMC) is configured by a profiling tool to count a particular
846 type of event. When the counter overflows, an interrupt is delivered to the processor.
847 This is the basic mechanism on which OProfile is based. The delivery mode is <acronym>NMI</acronym>,
848 so blocking interrupts in the kernel does not prevent profiling. When the interrupt handler is called,
849 the current <acronym>PC</acronym> (program counter) value and the current task are recorded into the profiling structure.
850 This allows the overflow event to be attributed to a specific assembly instruction in a specific binary image.
851 OProfile receives this data (commonly referred to as a "sample") from the kernel and writes it to the sample files.
854 If we use an event such as <constant>CPU_CLK_UNHALTED</constant> or <constant>INST_RETIRED</constant>
855 (<constant>GLOBAL_POWER_EVENTS</constant> or <constant>INSTR_RETIRED</constant>, respectively, on the Pentium 4), we can
856 use the overflow counts (samples) as an estimate of actual time spent in each part of code. Alternatively we can profile interesting
857 data such as the cache behaviour of routines with the other available counters.
860 However there are several caveats. First, there are those issues listed in the Intel manual. There is a delay
861 between the counter overflow and the interrupt delivery that can skew results on a small scale - this means
862 you cannot rely on the profiles at the instruction level as being perfectly accurate.
863 For example, if you are profiling an application with an event that counts L1 cache misses, a sample attributed
864 to a particular instruction in the application doesn't necessarily mean that exact instruction is responsible
865 for that event; instead, it means the sample was taken in the dynamic vicinity of that instruction,
866 usually with a margin of error of a few instructions. Further details on this problem can be found in
867 <xref linkend="interpreting" /> and also in the Digital paper "ProfileMe: A Hardware Performance Counter".
870 Each counter has several configuration parameters besides the type of event to count.
871 First, there is the unit mask, which is used to further qualify exactly what to count.
872 Second, there is the <constant>count</constant> field, discussed below. Third, there are parameters
873 to specify whether to increment counts
874 whilst in kernel or user space. You can configure these separately for each counter.
877 When the profiler is initially setup, a performance monitor counter is chosen for counting the
878 event, and it is initialized using the <constant>count</constant> value.
879 Once profiling begins, the counter increments with each event detected, and the counter
880 <emphasis>overflows</emphasis> when the <constant>count</constant> value is reached.
881 As described above, the counter overflow generates an interrupt, and the sample is recorded.
882 After each overflow event, the counter is re-initialized using the <constant>count</constant> value,
883 and counting begins anew for the next sample. Higher values for <constant>count</constant>
884 result in samples being taken less frequently, and therefore less-detailed (and, potentially,
885 less accurate) profiling. Lower values mean more detail, but higher overhead.
886 Picking a good value for this parameter is, unfortunately, somewhat of a black art. It is
887 of course dependent on the event you have chosen.
888 Specifying too large a value will mean not enough interrupts are generated
889 to give a realistic profile (though this problem can be ameliorated by profiling for
890 longer time periods. Specifying too small a value can lead to higher performance overhead.
896 <title>OProfile timer interrupt mode</title>
898 Some CPU types do not provide the needed hardware support for hardware performance counters.
899 Additionally, some older architectures are not supported by the perf_events kernel subsystem.
900 On such machines, the <command>operf</command> and <command>ocount</command> commands will exit with a message indicating the
901 processor type is not supported. However, you can install OProfile 0.9.9 and use the legacy
902 opcontrol-based profiler, which will fall back to using timer interrupts for profiling.
903 Note that in timer mode, OProfile is not able to profile code that has interrupts disabled.
904 <note>Timer mode is only available using the legacy <command>opcontrol</command> command,
905 available in releases prior to 1.0.</note>
909 <sect2 id="special-notes">
910 <title>Architecture-specific configuration notes</title>
912 <title>Pentium 4 support</title>
914 The Pentium 4 / Xeon performance counters are organized around 3 types of model specific registers (MSRs): 45 event
915 selection control registers (ESCRs), 18 counter configuration control registers (CCCRs) and 18 counters. ESCRs describe a
916 particular set of events which are to be recorded, and CCCRs bind ESCRs to counters and configure their
917 operation. Unfortunately the relationship between these registers is quite complex; they cannot all be used with one
918 another at any time. There is, however, a subset of 8 counters, 8 ESCRs, and 8 CCCRs which can be used independently of
919 one another, so OProfile only accesses those registers, treating them as a bank of 8 "normal" counters, similar
920 to those in the P6 or Athlon/Opteron/Phenom/Turion families of CPU.
923 There is currently no support for Precision Event-Based Sampling (PEBS), nor any advanced uses of the Debug Store
924 (DS). Current support is limited to the conservative extension of OProfile's existing interrupt-based model described
930 <title>PowerPC64 support</title>
932 The performance monitoring unit (PMU) for the IBM PowerPC 64-bit processors
933 consists of between 4 and 8 counters (depending on the model). Advanced features
934 such as instruction matching and thresholding are not supported by OProfile.
946 <chapter id="results">
947 <title>Obtaining profiling results</title>
949 After collecting profile data, the raw data must undergo special processing in order for you to
950 perform your analysis. The analysis tools that perform this special processing are
951 <command>opreport</command>, <command>opannotate</command>, and <command>opgprof</command>.
952 Additionally, the <command>oparchive</command> is used to gather together profile
953 data, sampled binary files, etc. for the purpose of off-line analysis. While
954 not really an analysis tool, <command>oparchive</command> is put in that category
955 for convenience since it takes many of the same options as the other analysis tools.
958 <sect1 id="profile-spec">
959 <title>Profile specifications</title>
962 All of the analysis tools take a <emphasis>profile specification</emphasis>
963 as an input argument.
964 This is a set of definitions that describes the specific profile data that should be
965 examined. The simplest profile specification is empty: this will match all
966 the available profile files for the current session.
969 Specification parameters are of the form <option>name:value[,value]</option>.
970 For example, if I wanted to get a combined symbol summary for
971 <filename>/bin/myprog</filename> and <filename>/bin/myprog2</filename>,
972 I could do <command>opreport -l image:/bin/myprog,/bin/myprog2</command>.
973 As a special case, you don't actually need to specify the <option>image:</option>
974 part of the specification. Anything left on the command line after all other
975 <command>opreport</command> options have been processed is assumed to be an
976 <option>image:</option> name. Similarly, if no <option>session:</option>
977 is specified, then <option>session:current</option> is assumed ("current"
978 is a special name of the current (i.e., most recent) profiling session).
981 In addition to the comma-separated list shown above, some of the
982 specification parameters can take <command>glob</command>-style
983 values. For example, if I want to see image summaries for all
984 binaries profiled in <filename>/usr/bin/</filename>, I could do
985 <command>opreport image:/usr/bin/\*</command>. Note the necessity
986 to escape the special character from the shell.
989 For <command>opreport</command>, profile specifications can be used to
990 define two profiles, giving differential output. This is done by
991 enclosing each of the two specifications within curly braces, as shown
992 in the examples below. Any specifications outside of curly braces are
996 <sect2 id="profile-spec-examples">
997 <title>Examples</title>
1000 Image summaries for all profiles with <constant>DATA_MEM_REFS</constant>
1001 samples in the saved session called "stresstest" :
1004 # opreport session:stresstest event:DATA_MEM_REFS
1008 Symbol summary for the application called "test_sym53c8xx,9xx". Note the
1009 escaping is necessary as <option>image:</option> takes a comma-separated list.
1012 # opreport -l ./test/test_sym53c8xx\,9xx
1016 Image summaries for all binaries in the <filename>test</filename> directory,
1017 excepting <filename>boring-test</filename> :
1020 # opreport image:./test/\* image-exclude:./test/boring-test
1024 Differential profile of a binary stored in two archives :
1027 # opreport -l /bin/bash { archive:./orig } { archive:./new }
1031 Differential profile of an archived binary with the current session :
1034 # opreport -l /bin/bash { archive:./orig } { }
1037 </sect2> <!-- profile spec examples -->
1039 <sect2 id="profile-spec-details">
1040 <title>Profile specification parameters</title>
1044 <term><option>archive:</option><emphasis>archivepath</emphasis></term>
1046 A path to an archive made with <command>oparchive</command>.
1047 Absence of this tag, unlike others, means "the current system",
1048 equivalent to specifying "archive:".
1052 <term><option>session:</option><emphasis>sessionlist</emphasis></term>
1054 A comma-separated list of session names to resolve in. Absence of this
1055 tag, unlike others, means "the current session", equivalent to
1056 specifying "session:current".
1060 <term><option>session-exclude:</option><emphasis>sessionlist</emphasis></term>
1062 A comma-separated list of sessions to exclude.
1066 <term><option>image:</option><emphasis>imagelist</emphasis></term>
1068 A comma-separated list of image names to resolve. Each entry may be relative
1069 path, <command>glob</command>-style name, or full path, e.g.</para>
1070 <screen>opreport 'image:/usr/bin/oprofiled,*op*,./opreport'</screen>
1075 <term><option>image-exclude:</option><emphasis>imagelist</emphasis></term>
1077 Same as <option>image:</option>, but the matching images are excluded.
1082 <term><option>lib-image:</option><emphasis>imagelist</emphasis></term>
1084 Same as <option>image:</option>, but only for images that are for
1085 a particular primary binary image (namely, an application).
1090 <term><option>lib-image-exclude:</option><emphasis>imagelist</emphasis></term>
1092 Same as <option>lib-image:</option>, but the matching images
1098 <term><option>event:</option><emphasis>eventlist</emphasis></term>
1100 The symbolic event name to match on, e.g. <option>event:DATA_MEM_REFS</option>.
1101 You can pass a list of events for side-by-side comparison with <command>opreport</command>.
1106 <term><option>count:</option><emphasis>eventcountlist</emphasis></term>
1108 The event count to match on, e.g. <option>event:DATA_MEM_REFS count:30000</option>.
1109 Note that this value refers to the count value in the event spec you passed
1110 to <command>operf</command> when setting up to do a
1111 profile run. It has nothing to do with the sample counts in the profile data
1113 You can pass a list of events for side-by-side comparison with <command>opreport</command>.
1118 <term><option>unit-mask:</option><emphasis>masklist</emphasis></term>
1120 The unit mask value of the event to match on, e.g. <option>unit-mask:1</option>.
1121 You can pass a list of events for side-by-side comparison with <command>opreport</command>.
1126 <term><option>cpu:</option><emphasis>cpulist</emphasis></term>
1128 Only consider profiles for the given numbered CPU (starting from zero).
1129 This is only useful when using CPU profile separation.
1134 <term><option>tgid:</option><emphasis>pidlist</emphasis></term>
1136 Only consider profiles for the given task groups. Unless some program
1137 is using threads, the task group ID of a process is the same
1138 as its process ID. This option corresponds to the POSIX
1139 notion of a thread group.
1140 This is only useful when using per-process profile separation.
1145 <term><option>tid:</option><emphasis>tidlist</emphasis></term>
1147 Only consider profiles for the given threads. When using
1148 recent thread libraries, all threads in a process share the
1149 same task group ID, but have different thread IDs. You can
1150 use this option in combination with <option>tgid:</option> to
1151 restrict the results to particular threads within a process.
1152 This is only useful when using per-process profile separation.
1159 <sect2 id="locating-and-managing-binary-images">
1160 <title>Locating and managing binary images</title>
1162 Each session's sample files can be found in the $SESSION_DIR/samples/ directory (default
1163 for <command>operf</command> is <filename><cur_dir>/oprofile_data/samples/</filename>).
1164 These are used, along with the binary image files, to produce human-readable data.
1165 In some circumstances (e.g., kernel modules), OProfile
1166 will not be able to find the binary images. All the tools have an <option>--image-path</option>
1167 option to which you can pass a comma-separated list of alternate paths to search. For example,
1168 I can let OProfile find my 2.6 modules by using <command>--image-path /lib/modules/2.6.0/kernel/</command>.
1169 It is your responsibility to ensure that the correct images are found when using this
1173 Note that if a binary image changes after the sample file was created, you won't be able to get useful
1174 symbol-based data out. This situation is detected for you. If you replace a binary, you should
1175 make sure to save the old binary if you need to do comparative profiles.
1180 <sect2 id="no-results">
1181 <title>What to do when you don't get any results</title>
1183 When attempting to get output, you may see the error :
1186 error: no sample files found: profile specification too strict ?
1189 What this is saying is that the profile specification you passed in,
1190 when matched against the available sample files, resulted in no matches.
1191 There are a number of reasons this might happen:
1194 <varlistentry><term>spelling</term><listitem><para>
1195 You specified a binary name, but spelt it wrongly. Check your spelling !
1196 </para></listitem></varlistentry>
1197 <varlistentry><term>profiler wasn't running</term><listitem><para>
1198 Make very sure that OProfile was actually up and running when you ran
1199 the application you wish to profile.
1200 </para></listitem></varlistentry>
1201 <varlistentry><term>application didn't run long enough</term><listitem><para>
1202 Remember OProfile is a statistical profiler - you're not guaranteed to
1203 get samples for short-running programs. You can help this by using a
1204 lower count for the performance counter, so there are a lot more samples
1206 </para></listitem></varlistentry>
1207 <varlistentry><term>application spent most of its time in libraries</term><listitem><para>
1208 Similarly, if the application spends little time in the main binary image
1209 itself, with most of it spent in shared libraries it uses, you might
1210 not see any samples for the binary image (i.e., executable) itself.
1211 </para></listitem></varlistentry>
1212 <varlistentry><term>specification was really too strict</term><listitem><para>
1213 For example, you specified something like <option>tgid:3433</option>,
1214 but no task with that group ID ever ran the code.
1215 </para></listitem></varlistentry>
1216 <varlistentry><term>application didn't generate any events</term><listitem><para>
1217 If you're profiling a particular event, for example counting MMX
1218 operations, the code might simply have not generated any events in the
1219 first place. Verify the code you're profiling does what you expect it
1221 </para></listitem></varlistentry>
1222 <varlistentry><term>you didn't specify kernel module name correctly</term><listitem><para>
1223 If you're trying to get reports for a kernel
1224 module, make sure to use the <option>-p</option> option, and specify the
1225 module name <emphasis>with</emphasis> the <filename>.ko</filename>
1226 extension. Check if the module is one loaded from initrd.
1227 </para></listitem></varlistentry>
1232 </sect1> <!-- profile-spec -->
1234 <sect1 id="opreport">
1235 <title>Image summaries and symbol summaries (<command>opreport</command>)</title>
1237 The <command>opreport</command> utility is the primary utility you will use for
1238 getting formatted data out of OProfile. It produces two types of data: image summaries
1239 and symbol summaries. An image summary lists the number of samples for individual
1240 binary images such as libraries or applications. Symbol summaries provide per-symbol
1241 profile data. In the following truncated example, we see an image summary for the whole
1245 $ opreport --long-filenames
1246 CPU: Intel Sandy Bridge microarchitecture, speed 2401 MHz (estimated)
1247 Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (No unit mask) count 100000
1251 22577 28.9011 /usr/bin/Xorg
1255 16846 74.6158 /proc/kallsyms
1256 2126 9.4167 /usr/bin/Xorg
1257 763 3.3795 /usr/lib64/libpixman-1.so.0.26.2
1259 17402 22.2766 /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.55.x86_64/jre/bin/java
1263 5666 32.5595 anon (tgid:29664 range:0x7f3475000000-0x7f347616ffff)
1264 2312 13.2858 /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.55.x86_64/jre/lib/amd64/server/libjvm.so
1266 11554 14.7904 /home/user1/oprof-install/bin/operf
1270 7467 64.6270 /proc/kallsyms
1271 1691 14.6356 /usr/bin/operf
1272 1324 11.4592 /lib64/libc-2.12.so
1273 455 3.9380 /usr/lib64/libstdc++.so.6.0.13
1279 If we had specified <option>--symbols</option> in the previous command, we would have
1280 gotten a symbol summary of all the images across the entire system. We can restrict this to only
1281 part of the system profile; for example,
1282 below is a symbol summary for the <command>operf</command> program used to collect the profile.
1285 $ opreport -l -p /lib/modules/`uname -r` `which operf` 2>/dev/null | more
1286 CPU: Intel Sandy Bridge microarchitecture, speed 2401 MHz (estimated)
1287 Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (No unit mask) count 100000
1288 samples % image name symbol name
1289 860 7.4607 kallsyms avtab_search_node
1290 474 4.1121 operf OP_perf_utils::op_write_event(event_union*, unsigned long long)
1291 461 3.9993 kallsyms avc_has_perm_noaudit
1292 455 3.9473 libstdc++.so.6.0.13 /usr/lib64/libstdc++.so.6.0.13
1293 412 3.5742 libc-2.12.so _IO_vfscanf
1294 369 3.2012 kallsyms __d_lookup
1295 350 3.0363 kallsyms sidtab_context_to_sid
1296 274 2.3770 operf OP_perf_utils::op_record_process_exec_mmaps(int, int, int, operf_record*)
1297 232 2.0127 operf operf_process_info::find_mapping_for_sample(unsigned long long, bool)
1298 222 1.9259 kallsyms __link_path_walk
1299 191 1.6570 kallsyms pipe_read
1300 34 0.2950 ext4.ko ext4_mark_iloc_dirty
1305 These are the two basic ways you are most likely to use regularly, but <command>opreport</command>
1306 can do a lot more than that, as described below.
1309 <sect2 id="opreport-merging">
1310 <title>Merging separate profiles</title>
1312 If you have used one of the <option>--separate[*]</option> options
1313 whilst profiling, there can be several separate profiles for
1314 a single binary image within a session. Normally the output
1315 will keep these images separated. So, for example, if you profiled
1316 with separation on a per-cpu basis (<code>operf --separate-cpu</code>),
1317 you would see separate columns in
1318 the output of <command>opreport</command> for each CPU where samples
1319 were recorded. But it can be useful to merge these results back together
1320 to make the report more readable. The <option>--merge</option> option allows
1324 <sect2 id="opreport-comparison">
1325 <title>Side-by-side multiple results</title>
1326 If you have used multiple events when profiling, by default you get
1327 side-by-side results of each event's sample values from <command>opreport</command>.
1328 You can restrict which events to list by appropriate use of the
1329 <option>event:</option> profile specifications, etc.
1332 <sect2 id="opreport-callgraph">
1333 <title>Callgraph output</title>
1335 This section provides details on how to use the OProfile callgraph feature.
1338 <title>Callgraph details</title>
1340 When using the <option>--callgraph</option> option, you can see what
1341 functions are calling other functions in the output. Consider the
1345 #include <string.h>
1346 #include <stdlib.h>
1347 #include <stdio.h>
1351 static int compare(const void *s1, const void *s2)
1353 return strcmp(s1, s2);
1356 static void repeat(void)
1359 char *strings[SIZE];
1360 char str[] = "abcdefghijklmnopqrstuvwxyz";
1362 for (i = 0; i < SIZE; ++i) {
1363 strings[i] = strdup(str);
1367 qsort(strings, SIZE, sizeof(char *), compare);
1377 When running with the call-graph option, OProfile will
1378 record the function stack every time it takes a sample.
1379 <command>opreport --callgraph</command> outputs an entry for each
1380 function, where each entry looks similar to:
1383 samples % image name symbol name
1385 127036 99.8452 cg repeat
1386 84590 42.5084 libc-2.3.2.so strfry
1387 84590 66.4838 libc-2.3.2.so strfry [self]
1388 39169 30.7850 libc-2.3.2.so random_r
1389 3475 2.7312 libc-2.3.2.so __i686.get_pc_thunk.bx
1390 -------------------------------------------------------------------------------
1393 Here the non-indented line is the function we're focussing upon
1394 (<function>strfry()</function>). This
1395 line is the same as you'd get from a normal <command>opreport</command>
1399 Above the non-indented line we find the functions that called this
1400 function (for example, <function>repeat()</function> calls
1401 <function>strfry()</function>). The samples and percentage values here
1402 refer to the number of times we took a sample where this call was found
1403 in the stack; the percentage is relative to all other callers of the
1404 function we're focussing on. Note that these values are
1405 <emphasis>not</emphasis> call counts; they only reflect the call stack
1406 every time a sample is taken; that is, if a call is found in the stack
1407 at the time of a sample, it is recorded in this count.
1410 Below the line are functions that are called by
1411 <function>strfry()</function> (called <emphasis>callees</emphasis>).
1412 It's clear here that <function>strfry()</function> calls
1413 <function>random_r()</function>. We also see a special entry with a
1414 "[self]" marker. This records the normal samples for the function, but
1415 the percentage becomes relative to all callees. This allows you to
1416 compare time spent in the function itself compared to functions it
1417 calls. Note that if a function calls itself, then it will appear in the
1418 list of callees of itself, but without the "[self]" marker; so recursive
1419 calls are still clearly separable.
1422 You may have noticed that the output lists <function>main()</function>
1423 as calling <function>strfry()</function>, but it's clear from the source
1424 that this doesn't actually happen. See <xref
1425 linkend="interpreting-callgraph" /> for an explanation.
1428 <sect3 id="cg-with-jitsupport">
1429 <title>Callgraph is not supported with JIT samples</title>
1431 Callgraph output where anonymously mapped code is in the callstack can sometimes be misleading.
1432 For all such code, the samples for the anonymously mapped code are stored in a samples subdirectory
1433 named <filename>{anon:anon}/<tgid>.<begin_addr>.<end_addr></filename>.
1434 As stated earlier, if this anonymously mapped code is JITed code from a supported VM like Java,
1435 OProfile creates an ELF file to provide a (somewhat) permanent backing file for the code.
1436 However, when viewing callgraph output, any anonymously mapped code in the callstack
1437 will be attributed to <filename>anon (<tgid>: range:<begin_addr>-<end_addr></filename>,
1438 even if a <filename>.jo</filename> ELF file had been created for it. See the example below.
1441 -------------------------------------------------------------------------------
1442 1 2.2727 libj9ute23.so java.bin traceV
1443 2 4.5455 libj9ute23.so java.bin utsTraceV
1444 4 9.0909 libj9trc23.so java.bin fillInUTInterfaces
1445 37 84.0909 libj9trc23.so java.bin twGetSequenceCounter
1446 8 0.0154 libj9prt23.so java.bin j9time_hires_clock
1447 27 61.3636 anon (tgid:10014 range:0x100000-0x103000) java.bin (no symbols)
1448 9 20.4545 libc-2.4.so java.bin gettimeofday
1449 8 18.1818 libj9prt23.so java.bin j9time_hires_clock [self]
1450 -------------------------------------------------------------------------------
1453 The output shows that "anon (tgid:10014 range:0x100000-0x103000)" was a callee of
1454 <code>j9time_hires_clock</code>, even though the ELF file <filename>10014.jo</filename> was
1455 created for this profile run. Unfortunately, there is currently no way to correlate
1456 that anonymous callgraph entry with its corresponding <filename>.jo</filename> file.
1461 </sect2> <!-- opreport-callgraph -->
1463 <sect2 id="opreport-diff">
1464 <title>Differential profiles with <command>opreport</command></title>
1467 Often, we'd like to be able to compare two profiles. For example, when
1468 analysing the performance of an application, we'd like to make code
1469 changes and examine the effect of the change. This is supported in
1470 <command>opreport</command> by giving a profile specification that
1471 identifies two different profiles. The general form is of:
1474 $ opreport <shared-spec> { <first-profile> } { <second-profile> }
1477 We lost our Dragon book down the back of the sofa, so you have to be
1478 careful to have spaces around those braces, or things will get
1479 hopelessly confused. We can only apologise.
1482 For each of the profiles, the shared section is prefixed, and then the
1483 specification is analysed. The usual parameters work both within the
1484 shared section, and in the sub-specification within the curly braces.
1487 A typical way to use this feature is with archives created with
1488 <command>oparchive</command>. Let's look at an example:
1492 $ oparchive -o orig ./a
1493 # edit and recompile a
1495 # now compare the current profile of a with the archived profile
1496 $ opreport --session-dir=`pwd`/oprofile_data/ -xl ./a { archive:./orig } { }
1497 CPU: PIII, speed 863.233 MHz (estimated)
1498 Counted CPU_CLK_UNHALTED events (clocks processor is not halted) with a
1499 unit mask of 0x00 (No unit mask) count 100000
1500 samples % diff % symbol name
1501 92435 48.5366 +0.4999 a
1504 48787 25.6175 -2.2e-01 b
1507 Note that we specified an empty second profile in the curly braces, as
1508 we wanted to use the current session; alternatively, we could
1509 have specified another archive, or a tgid etc. We specified the binary
1510 <command>a</command> in the shared section, so we matched that in both
1511 the profiles we're diffing.
1514 As in the normal output, the results are sorted by the number of
1515 samples, and the percentage field represents the relative percentage of
1516 the symbol's samples in the second profile.
1519 Notice the new column in the output. This value represents the
1520 percentage change of the relative percent between the first and the
1521 second profile: roughly, "how much more important this symbol is".
1522 Looking at the symbol <function>a()</function>, we can see that it took
1523 roughly the same amount of the total profile in both the first and the
1524 second profile. The function <function>c()</function> was not in the new
1525 profile, so has been marked with <function>---</function>. Note that the
1526 sample value is the number of samples in the first profile; since we're
1527 displaying results for the second profile, we don't list a percentage
1528 value for it, as it would be meaningless. <function>d()</function> is
1529 new in the second profile, and consequently marked with
1530 <function>+++</function>.
1533 When comparing profiles between different binaries, it should be clear
1534 that functions can change in terms of VMA and size. To avoid this
1535 problem, <command>opreport</command> considers a symbol to be the same
1536 if the symbol name, image name, and owning application name all match;
1537 any other factors are ignored. Note that the check for application name
1538 means that trying to compare library profiles between two different
1539 applications will not work as you might expect: each symbol will be
1540 considered different.
1543 </sect2> <!-- opreport-diff -->
1545 <sect2 id="opreport-anon">
1546 <title>Anonymous executable mappings</title>
1548 Many applications, typically ones involving dynamic compilation into
1549 machine code (just-in-time, or "JIT", compilation), have executable mappings that
1550 are not backed by an ELF file. <command>opreport</command> has basic support for showing the
1551 samples taken in these regions; for example:
1553 $ opreport /usr/bin/mono -l
1554 CPU: ppc64 POWER5, speed 1654.34 MHz (estimated)
1555 Counted CYCLES events (Processor Cycles using continuous sampling) with a unit mask of 0x00 (No unit mask) count 100000
1556 samples % image name symbol name
1557 47 58.7500 mono (no symbols)
1558 14 17.5000 anon (tgid:3189 range:0xf72aa000-0xf72fa000) (no symbols)
1559 9 11.2500 anon (tgid:3189 range:0xf6cca000-0xf6dd9000) (no symbols)
1564 Note that, since such mappings are dependent upon individual invocations of
1565 a binary, these mappings are always listed as a dependent image.
1566 Equally, the results are not affected by the <option>--merge</option>
1570 As shown in the opreport output above, OProfile is unable to attribute the samples to any
1571 symbol(s) because there is no ELF file for this code.
1572 Enhanced support for JITed code is now available for some virtual machines;
1573 e.g., the Java Virtual Machine. For details about OProfile output for
1574 JITed code, see <xref linkend="getting-jit-reports" />.
1576 <para>For more information about JIT support in OProfile, see <xref linkend="jitsupport"/>.
1578 </sect2> <!-- opreport-anon -->
1580 <sect2 id="opreport-xml">
1581 <title>XML formatted output</title>
1583 The --xml option can be used to generate XML instead of the usual
1584 text format. This allows opreport to eliminate some of the constraints
1585 dictated by the two dimensional text format. For example, it is possible
1586 to separate the sample data across multiple events, cpus and threads. The XML
1587 schema implemented by opreport is found in doc/opreport.xsd. It contains
1588 more detailed comments about the structure of the XML generated by opreport.
1591 Since XML is consumed by a client program rather than a user, its structure
1592 is fairly static. In particular, the --sort option is incompatible with the
1593 --xml option. Percentages are not dislayed in the XML so the options related
1594 to percentages will have no effect. Full pathnames are always displayed in
1595 the XML so --long-filenames is not necessary. The --details option will cause
1596 all of the individual sample data to be included in the XML as well as the
1597 instruction byte stream for each symbol (for doing disassembly) and can result
1598 in very large XML files.
1600 </sect2> <!-- opreport-xml -->
1602 <sect2 id="opreport-options">
1603 <title>Options for <command>opreport</command></title>
1606 <varlistentry><term><option>--accumulated / -a</option></term><listitem><para>
1607 Accumulate sample and percentage counts in the symbol list.
1608 </para></listitem></varlistentry>
1609 <varlistentry><term><option>--callgraph / -c</option></term><listitem><para>
1610 Show callgraph information.
1611 </para></listitem></varlistentry>
1612 <varlistentry><term><option>--debug-info / -g</option></term><listitem><para>
1613 Show source file and line for each symbol.
1614 </para></listitem></varlistentry>
1615 <varlistentry><term><option>--demangle / -D none|normal|smart</option></term><listitem><para>
1616 none: no demangling. normal: use default demangler (default) smart: use
1617 pattern-matching to make C++ symbol demangling more readable.
1618 </para></listitem></varlistentry>
1619 <varlistentry><term><option>--details / -d</option></term><listitem><para>
1620 Show per-instruction details for all selected symbols. Note that, for
1621 binaries without symbol information, the VMA values shown are raw file
1622 offsets for the image binary.
1623 </para></listitem></varlistentry>
1624 <varlistentry><term><option>--exclude-dependent / -x</option></term><listitem><para>
1625 Do not include application-specific images for libraries, kernel modules
1627 </para></listitem></varlistentry>
1628 <varlistentry><term><option>--exclude-symbols / -e [symbols]</option></term><listitem><para>
1629 Exclude all the symbols in the given comma-separated list.
1630 </para></listitem></varlistentry>
1631 <varlistentry><term><option>--global-percent / -%</option></term><listitem><para>
1632 Make all percentages relative to the whole profile.
1633 </para></listitem></varlistentry>
1634 <varlistentry><term><option>--help / -? / --usage</option></term><listitem><para>
1636 </para></listitem></varlistentry>
1637 <varlistentry><term><option>--image-path / -p [paths]</option></term><listitem><para>
1638 Comma-separated list of additional paths to search for binaries.
1639 This is needed to find kernel modules.
1640 </para></listitem></varlistentry>
1641 <varlistentry><term><option>--root / -R [path]</option></term><listitem><para>
1642 A path to a filesystem to search for additional binaries.
1643 </para></listitem></varlistentry>
1644 <varlistentry><term><option>--include-symbols / -i [symbols]</option></term><listitem><para>
1645 Only include symbols in the given comma-separated list.
1646 </para></listitem></varlistentry>
1647 <varlistentry><term><option>--long-filenames / -f</option></term><listitem><para>
1648 Output full paths instead of basenames.
1649 </para></listitem></varlistentry>
1650 <varlistentry><term><option>--merge / -m [lib,cpu,tid,tgid,unitmask,all]</option></term><listitem><para>
1651 Merge any profiles separated in a --separate session.
1652 </para></listitem></varlistentry>
1653 <varlistentry><term><option>--no-header</option></term><listitem><para>
1654 Don't output a header detailing profiling parameters.
1655 </para></listitem></varlistentry>
1656 <varlistentry><term><option>--output-file / -o [file]</option></term><listitem><para>
1657 Output to the given file instead of stdout.
1658 </para></listitem></varlistentry>
1659 <varlistentry><term><option>--reverse-sort / -r</option></term><listitem><para>
1660 Reverse the sort from the default.
1661 </para></listitem></varlistentry>
1662 <varlistentry><term><option>--session-dir=dir_path</option></term><listitem><para>
1663 Use sample database from the specified directory <filename>dir_path</filename> instead
1664 of the default location. If this option is not specified, then opreport will search for
1665 samples in <filename><cur_dir>/oprofile_data</filename>
1666 first. If that directory does not exist, the standard session-dir of
1667 <filename>/var/lib/oprofile</filename> is used
1668 as the session directory.
1669 </para></listitem></varlistentry>
1670 <varlistentry><term><option>--show-address / -w</option></term><listitem><para>
1671 Show the VMA address of each symbol (off by default).
1672 </para></listitem></varlistentry>
1673 <varlistentry><term><option>--sort / -s [vma,sample,symbol,debug,image]</option></term><listitem><para>
1674 Sort the list of symbols by, respectively, symbol address,
1675 number of samples, symbol name, debug filename and line number,
1676 binary image filename.
1677 </para></listitem></varlistentry>
1678 <varlistentry><term><option>--symbols / -l</option></term><listitem><para>
1679 List per-symbol information instead of a binary image summary.
1680 </para></listitem></varlistentry>
1681 <varlistentry><term><option>--threshold / -t [percentage]</option></term><listitem><para>
1682 Only output data for symbols that have more than the given percentage
1683 of total samples. For profiles using multiple events, if the threshold is reached
1684 for any event, then all sample data for the symbol is shown.
1685 </para></listitem></varlistentry>
1686 <varlistentry><term><option>--verbose / -V [options]</option></term><listitem><para>
1687 Give verbose debugging output.
1688 </para></listitem></varlistentry>
1689 <varlistentry><term><option>--version / -v</option></term><listitem><para>
1691 </para></listitem></varlistentry>
1692 <varlistentry><term><option>--xml / -X</option></term><listitem><para>
1693 Generate XML output.
1694 </para></listitem></varlistentry>
1699 </sect1> <!-- opreport -->
1701 <sect1 id="opannotate">
1702 <title>Outputting annotated source (<command>opannotate</command>)</title>
1704 The <command>opannotate</command> utility generates annotated source files or assembly listings, optionally
1706 If you want to see the source file, the profiled application needs to have debug information, and the source
1707 must be available through this debug information. For GCC, you must use the <option>-g</option> option
1708 when you are compiling.
1709 If the binary doesn't contain sufficient debug information, you can still
1710 use <command>opannotate <option>--assembly</option></command> to get annotated assembly
1711 as long as the binary has (at least) symbol information.
1714 Note that for the reason explained in <xref linkend="hardware-counters" /> the results can be
1715 inaccurate. The debug information itself can add other problems; for example, the line number for a symbol can be
1716 incorrect. Assembly instructions can be re-ordered and moved by the compiler, and this can lead to
1717 crediting source lines with samples not really "owned" by this line. Also see
1718 <xref linkend="interpreting" />.
1721 You can output the annotation to one single file, containing all the source found using the
1722 <option>--source</option>. You can use this in conjunction with <option>--assembly</option>
1723 to get combined source/assembly output.
1726 You can also output a directory of annotated source files that maintains the structure of
1727 the original sources. Each line in the annotated source is prepended with the samples
1728 for that line. Additionally, each symbol is annotated giving details for the symbol
1729 as a whole. An example:
1732 $ opannotate --source --output-dir=annotated /usr/local/oprofile-pp/bin/oprofiled
1733 $ ls annotated/home/moz/src/oprofile-pp/daemon/
1734 opd_cookie.h opd_image.c opd_kernel.c opd_sample_files.c oprofiled.c
1737 Line numbers are maintained in the source files, but each file has
1738 a footer appended describing the profiling details. The actual annotation
1739 looks something like this :
1743 :static uint64_t pop_buffer_value(struct transient * trans)
1744 11510 1.9661 :{ /* pop_buffer_value total: 89901 15.3566 */
1747 10227 1.7469 : if (!trans->remaining) {
1748 : fprintf(stderr, "BUG: popping empty buffer !\n");
1749 : exit(EXIT_FAILURE);
1752 : val = get_buffer_value(trans->buffer, 0);
1753 2281 0.3896 : trans->remaining--;
1754 2296 0.3922 : trans->buffer += kernel_pointer_size;
1761 The first number on each line is the number of samples, whilst the second is
1762 the relative percentage of total samples.
1765 <sect2 id="opannotate-finding-source">
1766 <title>Locating source files</title>
1768 Of course, <command>opannotate</command> needs to be able to locate the source files
1769 for the binary image(s) in order to produce output. Some binary images have debug
1770 information where the given source file paths are relative, not absolute. You can
1771 specify search paths to look for these files (similar to <command>gdb</command>'s
1772 <option>dir</option> command) with the <option>--search-dirs</option> option.
1775 Sometimes you may have a binary image which gives absolute paths for the source files,
1776 but you have the actual sources elsewhere (commonly, you've installed an SRPM for
1777 a binary on your system and you want annotation from an existing profile). You can
1778 use the <option>--base-dirs</option> option to redirect OProfile to look somewhere
1779 else for source files. For example, imagine we have a binary generated from a source
1780 file that is given in the debug information as <filename>/tmp/build/libfoo/foo.c</filename>,
1781 and you have the source tree matching that binary installed in <filename>/home/user/libfoo/</filename>.
1782 You can redirect OProfile to find <filename>foo.c</filename> correctly like this :
1785 $ opannotate --source --base-dirs=/tmp/build/libfoo/ --search-dirs=/home/user/libfoo/ --output-dir=annotated/ /lib/libfoo.so
1788 You can specify multiple (comma-separated) paths to both options.
1792 <sect2 id="opannotate-details">
1793 <title>Usage of <command>opannotate</command></title>
1796 <varlistentry><term><option>--assembly / -a</option></term><listitem><para>
1797 Output annotated assembly. If this is combined with --source, then mixed
1798 source / assembly annotations are output.
1799 </para></listitem></varlistentry>
1800 <varlistentry><term><option>--base-dirs / -b [paths]/</option></term><listitem><para>
1801 Comma-separated list of path prefixes. This can be used to point OProfile to a
1802 different location for source files when the debug information specifies an
1803 absolute path on your system for the source that does not exist. The prefix
1804 is stripped from the debug source file paths, then searched in the search dirs
1805 specified by <option>--search-dirs</option>.
1806 </para></listitem></varlistentry>
1807 <varlistentry><term><option>--demangle / -D none|normal|smart</option></term><listitem><para>
1808 none: no demangling. normal: use default demangler (default) smart: use
1809 pattern-matching to make C++ symbol demangling more readable.
1810 </para></listitem></varlistentry>
1811 <varlistentry><term><option>--exclude-dependent / -x</option></term><listitem><para>
1812 Do not include application-specific images for libraries, kernel modules
1814 </para></listitem></varlistentry>
1815 <varlistentry><term><option>--exclude-file [files]</option></term><listitem><para>
1816 Exclude all files in the given comma-separated list of glob patterns.
1817 This option is supported solely with the <code>--source</code>
1818 option. It can be used to filter out source files in the output using the
1819 following types of specifications:
1821 <listitem>filenames (basename -- i.e., no path)</listitem>
1822 <listitem>filename glob specifications (all files whose base filename matches the given pattern)</listitem>
1823 <listitem>directory segments (all source files located in the specified directory; e.g. "libio")</listitem>
1824 <listitem>directory segment glob specifications (e.g., "libi*")</listitem>
1826 </para></listitem></varlistentry>
1827 <varlistentry><term><option>--exclude-symbols / -e [symbols]</option></term><listitem><para>
1828 Exclude all the symbols in the given comma-separated list.
1829 </para></listitem></varlistentry>
1830 <varlistentry><term><option>--help / -? / --usage</option></term><listitem><para>
1832 </para></listitem></varlistentry>
1833 <varlistentry><term><option>--image-path / -p [paths]</option></term><listitem><para>
1834 Comma-separated list of additional paths to search for binaries.
1835 This is needed to find kernel modules.
1836 </para></listitem></varlistentry>
1837 <varlistentry><term><option>--root / -R [path]</option></term><listitem><para>
1838 A path to a filesystem to search for additional binaries.
1839 </para></listitem></varlistentry>
1840 <varlistentry><term><option>--include-file [files]</option></term><listitem><para>
1841 Only include files in the given comma-separated list of glob patterns.
1842 The same rules apply for this option as for the <code>--exclude-file</code> option.
1843 </para></listitem></varlistentry>
1844 <varlistentry><term><option>--include-symbols / -i [symbols]</option></term><listitem><para>
1845 Only include symbols in the given comma-separated list.
1846 </para></listitem></varlistentry>
1847 <varlistentry><term><option>--objdump-params [params]</option></term><listitem><para>
1848 Pass the given parameters as extra values when calling objdump.
1849 If more than one option is to be passed to objdump, the parameters must be enclosed in a
1853 An example of where this option is useful is when your toolchain does not
1854 automatically recognize instructions that are specific to your processor.
1855 For example, on IBM POWER7/RHEL 6, objdump must be told that a binary file may have
1856 POWER7-specific instructions. The <command>opannotate</command> option to show the POWER7-specific
1859 --objdump-params=-Mpower7
1863 The <command>opannotate</command> option to show the POWER7-specific instructions,
1864 the source code (--source) and the line numbers (-l) would be:
1866 --objdump-params="-Mpower7 -l --source"
1868 </para></listitem></varlistentry>
1869 <varlistentry><term><option>--output-dir / -o [dir]</option></term><listitem><para>
1870 Output directory. This makes opannotate output one annotated file for each
1871 source file. This option can't be used in conjunction with --assembly.
1872 </para></listitem></varlistentry>
1873 <varlistentry><term><option>--search-dirs / -d [paths]</option></term><listitem><para>
1874 Comma-separated list of paths to search for source files. This is useful to find
1875 source files when the debug information only contains relative paths.
1876 </para></listitem></varlistentry>
1877 <varlistentry><term><option>--source / -s</option></term><listitem><para>
1878 Output annotated source. This requires debugging information to be available
1880 </para></listitem></varlistentry>
1881 <varlistentry><term><option>--session-dir=dir_path</option></term><listitem><para>
1882 Use sample database from the specified directory <filename>dir_path</filename> instead
1883 of the default location. If this option is not specified, then opannotate will search for
1884 samples in <filename><cur_dir>/oprofile_data</filename>
1885 first. If that directory does not exist, the standard session-dir of
1886 <filename>/var/lib/oprofile</filename> is used
1887 as the session directory.
1888 </para></listitem></varlistentry>
1889 <varlistentry><term><option>--threshold / -t [percentage]</option></term><listitem><para>
1890 For annotated assembly, only output data for symbols that have more than the given percentage
1891 of total samples. For profiles using multiple events, if the threshold is reached
1892 for any event, then all sample data for the symbol is shown.
1895 For annotated source, only output data for source files that have more than the given percentage
1896 of total samples. For profiles using multiple events, if the threshold is reached
1897 for any event, then all sample data for the source file is shown.
1898 </para></listitem></varlistentry>
1899 <varlistentry><term><option>--verbose / -V [options]</option></term><listitem><para>
1900 Give verbose debugging output.
1901 </para></listitem></varlistentry>
1902 <varlistentry><term><option>--version / -v</option></term><listitem><para>
1904 </para></listitem></varlistentry>
1908 </sect2> <!-- opannotate-details -->
1910 </sect1> <!-- opannotate -->
1912 <sect1 id="getting-jit-reports">
1913 <title>OProfile results with JIT samples</title>
1915 After profiling a Java (or other supported VM) application, the
1916 OProfile JIT support creates ELF binaries from the
1917 intermediate files that were written by the agent library.
1918 The ELF binaries are named <filename><tgid>.jo</filename>.
1919 With the symbol information stored in these ELF files, it is
1920 possible to map samples to the appropriate symbols.
1923 The usual analysis tools (<command>opreport</command> and/or
1924 <command>opannotate</command>) can now be used
1925 to get symbols and assembly code for the instrumented VM processes.
1928 Below is an example of a profile report of a Java application that has been
1929 instrumented with the provided agent library.
1931 $ opreport -l /usr/lib/jvm/jre-1.5.0-ibm/bin/java
1932 CPU: Core Solo / Duo, speed 2167 MHz (estimated)
1933 Counted CPU_CLK_UNHALTED events (Unhalted clock cycles) with a unit mask of 0x00 (Unhalted core cycles) count 100000
1934 samples % image name symbol name
1935 186020 50.0523 no-vmlinux no-vmlinux (no symbols)
1936 34333 9.2380 7635.jo java void test.f1()
1937 19022 5.1182 libc-2.5.so libc-2.5.so _IO_file_xsputn@@GLIBC_2.1
1938 18762 5.0483 libc-2.5.so libc-2.5.so vfprintf
1939 16408 4.4149 7635.jo java void test$HelloThread.run()
1940 16250 4.3724 7635.jo java void test$test_1.f2(int)
1941 15303 4.1176 7635.jo java void test.f2(int, int)
1942 13252 3.5657 7635.jo java void test.f2(int)
1943 5165 1.3897 7635.jo java void test.f4()
1944 955 0.2570 7635.jo java void test$HelloThread.run()~
1949 Depending on the JVM that is used, certain options of opreport and opannotate
1950 do NOT work since they rely on debug information (e.g. source code line number)
1951 that is not always available. The Sun JVM does provide the necessary debug
1952 information via the JVMTI[PI] interface,
1953 but other JVMs do not.
1956 As you can see in the opreport output, the JIT support agent for Java
1957 generates symbols to include the class and method signature.
1958 A symbol with the suffix ˜<n> (e.g.
1959 <code>void test$HelloThread.run()˜1</code>) means that this is
1960 the <n>th occurrence of the identical name. This happens if a method is re-JITed.
1961 A symbol with the suffix %<n>, means that the address space of this symbol
1962 was reused during the sample session (see <xref linkend="overlapping-symbols" />).
1963 The value <n> is the percentage of time that this symbol/code was present in
1964 relation to the total lifetime of all overlapping other symbols. A symbol of the form
1965 <code><return_val> <class_name>$<method_sig></code> denotes an
1970 <sect1 id="opgprof">
1971 <title><command>gprof</command>-compatible output (<command>opgprof</command>)</title>
1973 If you're familiar with the output produced by <command>GNU gprof</command>,
1974 you may find <command>opgprof</command> useful. It takes a single binary
1975 as an argument, and produces a <filename>gmon.out</filename> file for use
1976 with <command>gprof -p</command>. If call-graph profiling is enabled,
1977 then this is also included.
1980 $ opgprof `which oprofiled` # generates gmon.out file
1981 $ gprof -p `which oprofiled` | head
1984 Each sample counts as 1 samples.
1985 % cumulative self self total
1986 time samples samples calls T1/call T1/call name
1987 33.13 206237.00 206237.00 odb_insert
1988 22.67 347386.00 141149.00 pop_buffer_value
1989 9.56 406881.00 59495.00 opd_put_sample
1990 7.34 452599.00 45718.00 opd_find_image
1991 7.19 497327.00 44728.00 opd_process_samples
1994 <sect2 id="opgprof-details">
1995 <title>Usage of <command>opgprof</command></title>
1998 <varlistentry><term><option>--help / -? / --usage</option></term><listitem><para>
2000 </para></listitem></varlistentry>
2001 <varlistentry><term><option>--image-path / -p [paths]</option></term><listitem><para>
2002 Comma-separated list of additional paths to search for binaries.
2003 This is needed to find kernel modules.
2004 </para></listitem></varlistentry>
2005 <varlistentry><term><option>--root / -R [path]</option></term><listitem><para>
2006 A path to a filesystem to search for additional binaries.
2007 </para></listitem></varlistentry>
2008 <varlistentry><term><option>--output-filename / -o [file]</option></term><listitem><para>
2009 Output to the given file instead of the default, gmon.out
2010 </para></listitem></varlistentry>
2011 <varlistentry><term><option>--threshold / -t [percentage]</option></term><listitem><para>
2012 Only output data for symbols that have more than the given percentage
2014 </para></listitem></varlistentry>
2015 <varlistentry><term><option>--verbose / -V [options]</option></term><listitem><para>
2016 Give verbose debugging output.
2017 <varlistentry><term><option>--session-dir=dir_path</option></term><listitem><para>
2018 Use sample database from the specified directory <filename>dir_path</filename> instead
2019 of the default location. If this option is not specified, then opgprof will search for
2020 samples in <filename><cur_dir>/oprofile_data</filename>
2021 first. If that directory does not exist, the standard session-dir of
2022 <filename>/var/lib/oprofile</filename> is used
2023 as the session directory.
2024 </para></listitem></varlistentry>
2025 </para></listitem></varlistentry>
2026 <varlistentry><term><option>--version / -v</option></term><listitem><para>
2028 </para></listitem></varlistentry>
2031 </sect2> <!-- opgprof-details -->
2033 </sect1> <!-- opgprof -->
2035 <sect1 id="oparchive">
2036 <title>Analyzing profile data on another system (<command>oparchive</command>)</title>
2038 The <command>oparchive</command> utility generates a directory populated
2039 with executable, debug, and oprofile sample files. This directory can be
2040 copied to another (host) machine and analyzed offline, with no further need to
2041 access the data collection machine (target).
2045 The following command, executed on the target system, will collect the
2046 sample files, the executables associated with the sample files, and the
2047 debuginfo files associated with the executables and copy them into
2048 <filename>/tmp/current_data</filename>:
2052 # oparchive -o /tmp/current_data
2056 When transferring archived profile data to a host machine for offline analysis,
2057 you need to determine if the oprofile ABI format is compatible between the
2058 target system and the host system; if it isn't, you must run the <command>opimport</command>
2059 command to convert the target's sample data files to the format of your host system.
2060 See <xref linkend="opimport"/> for more details.
2064 After your profile data is transferred to the host system and (if necessary)
2065 you have run the <command>opimport</command> command to convert the file
2066 format, you can now run the <command>opreport</command> and
2067 <command>opannotate</command> commands. However, you must provide an
2068 "archive specification" to let these post-processing tools know where to find
2069 of the profile data (sample files, executables, etc.); for example:
2073 # opreport archive:/home/user1/my_oprofile_archive --symbols
2077 Furthermore, if your profile was collected on your target system into a session-dir
2078 other than <filename>/var/lib/oprofile</filename>, the <command>oparchive</command>
2079 command will display a message similar to the following:
2083 # NOTE: The sample data in this archive is located at /home/user1/test-stuff/oprofile_data
2084 instead of the standard location of /var/lib/oprofile. Hence, when using opreport
2085 and other post-processing tools on this archive, you must pass the following option:
2086 --session-dir=/home/user1/test-stuff/oprofile_data
2090 Then the above <command>opreport</command> example would have to include that
2091 <option>--session-dir</option> option.
2096 In some host/target development environments, all target executables, libraries, and
2097 debuginfo files are stored in a root directory on the host to facilitate offline
2098 analysis. In such cases, the <command>oparchive</command> command collects more data
2099 than is necessary; so, when copying the resulting output of <command>oparchive</command>,
2100 you can skip all of the executables, etc, and just archive the <filename>$SESSION_DIR</filename>
2101 tree located within the output directory you specified in your <command>oparchive</command>
2102 command. Then, when running the <command>opreport</command> or <command>opannotate</command>
2103 commands on your host system, pass the <option>--root</option> option to point to the
2104 location of your target's executables, etc.
2108 <sect2 id="oparchive-details">
2109 <title>Usage of <command>oparchive</command></title>
2112 <varlistentry><term><option>--help / -? / --usage</option></term><listitem><para>
2114 </para></listitem></varlistentry>
2115 <varlistentry><term><option>--exclude-dependent / -x</option></term><listitem><para>
2116 Do not include application-specific images for libraries, kernel modules
2118 </para></listitem></varlistentry>
2119 <varlistentry><term><option>--image-path / -p [paths]</option></term><listitem><para>
2120 Comma-separated list of additional paths to search for binaries.
2121 This is needed to find kernel modules.
2122 </para></listitem></varlistentry>
2123 <varlistentry><term><option>--root / -R [path]</option></term><listitem><para>
2124 A path to a filesystem to search for additional binaries.
2125 </para></listitem></varlistentry>
2126 <varlistentry><term><option>--output-directory / -o [directory]</option></term><listitem><para>
2127 Output to the given directory. There is no default. This must be specified.
2128 </para></listitem></varlistentry>
2129 <varlistentry><term><option>--list-files / -l</option></term><listitem><para>
2130 Only list the files that would be archived, don't copy them.
2131 </para></listitem></varlistentry>
2132 <varlistentry><term><option>--verbose / -V [options]</option></term><listitem><para>
2133 Give verbose debugging output.
2134 </para></listitem></varlistentry>
2135 <varlistentry><term><option>--session-dir=dir_path</option></term><listitem><para>
2136 Use sample database from the specified directory <filename>dir_path</filename> instead
2137 of the default location. If this option is not specified, then oparchive will search for
2138 samples in <filename><cur_dir>/oprofile_data</filename>
2139 first. If that directory does not exist, the standard session-dir of
2140 <filename>/var/lib/oprofile</filename> is used
2141 as the session directory.
2142 </para></listitem></varlistentry>
2143 <varlistentry><term><option>--version / -v</option></term><listitem><para>
2145 </para></listitem></varlistentry>
2148 </sect2> <!-- oparchive-details -->
2150 </sect1> <!-- oparchive -->
2152 <sect1 id="opimport">
2153 <title>Converting sample database files (<command>opimport</command>)</title>
2155 This utility converts sample database files from a foreign binary format (abi) to
2156 the native format. This is required when moving sample files to a (host) system
2157 other than the one used for collection (target system), and the host and target systems are different
2158 architectures. The abi format of the sample files to be imported is described in a
2159 text file located in <filename>$SESSION_DIR/abi</filename>. If you are unsure if
2160 your target and host systems have compatible architectures (in regard to the OProfile
2161 ABI), simply diff a <filename>$SESSION_DIR/abi</filename> file from the target system
2162 with one from the host system. If any differences show up at all, you must run the
2163 <command>opimport</command> command.
2167 The <command>oparchive</command> command should be used on the machine where
2168 the profile was taken (target) in order to collect sample files and all other necessary
2169 information. The archive directory that is the output from <command>oparchive</command>
2170 should be copied to the system where you wish to perform your performance analysis (host).
2174 The following command converts an input sample file to the specified
2175 output sample file using the given abi file as a binary description
2176 of the input file and the curent platform abi as a binary description
2177 of the output file. (NOTE: The ellipses are used to make the example more
2178 compact and cannot be used in an actual command line.)
2182 # opimport -a /tmp/foreign-abi -o /tmp/imported/.../GLOBAL_POWER_EVENTS.200000.1.all.all.all /tmp/archived/var/lib/.../mprime/GLOBAL_POWER_EVENTS.200000.1.all.all.all
2185 Since opimport converts just one file at a time, an example shell script is provided below
2186 that will perform an import/conversion of all sample files in a samples directory collected
2187 from the target system.
2190 Usage: my-import.sh <input-abi-pathname>
2192 # NOTE: Start from the "samples" directory containing the "current" directory
2195 mkdir current-imported
2196 cd current-imported; (cd ../current; find . -type d ! -name .) |xargs mkdir
2197 cd ../current; mv stats ../StatsSave; find . -type f | while read line; do opimport -a $1 -o ../current-imported/$line $line; done; mv ../StatsSave stats;
2201 Example usage: Assume that on the target system, a profile was collected using a session-dir of
2202 <filename>/var/lib/oprofile</filename>, and then <command>oparchive -o profile1</command> was run.
2203 Then the <filename>profile1</filename> directory is copied to the host system for analysis. To import
2204 the sample data in <filename>profile1</filename>, you would perform the following steps:
2206 $cd profile1/var/lib/oprofile/samples
2207 $my-import.sh `pwd`/../abi
2210 <sect2 id="opimport-details">
2211 <title>Usage of <command>opimport</command></title>
2214 <varlistentry><term><option>--help / -? / --usage</option></term><listitem><para>
2216 </para></listitem></varlistentry>
2217 <varlistentry><term><option>--abi / -a [filename]</option></term><listitem><para>
2218 Input abi file description location.
2219 </para></listitem></varlistentry>
2220 <varlistentry><term><option>--force / -f</option></term><listitem><para>
2221 Force conversion even if the input and output abi are identical.
2222 </para></listitem></varlistentry>
2223 <varlistentry><term><option>--output / -o [filename]</option></term><listitem><para>
2224 Specify the output filename. If the output file already exists, the file is
2225 not overwritten but data are accumulated in. Sample filename are informative
2226 for post profile tools and must be kept identical, in other word the pathname
2227 from the first path component containing a '{' must be kept as it in the
2229 </para></listitem></varlistentry>
2230 <varlistentry><term><option>--verbose / -V</option></term><listitem><para>
2231 Give verbose debugging output.
2232 </para></listitem></varlistentry>
2233 <varlistentry><term><option>--version / -v</option></term><listitem><para>
2235 </para></listitem></varlistentry>
2238 </sect2> <!-- opimport-details -->
2240 </sect1> <!-- opimport -->
2244 <chapter id="interpreting">
2245 <title>Interpreting profiling results</title>
2247 The standard caveats of profiling apply in interpreting the results from OProfile:
2248 profile realistic situations, profile different scenarios, profile
2249 for as long as a time as possible, avoid system-specific artifacts, don't trust
2250 the profile data too much. Also bear in mind the comments on the performance
2251 counters above - you <emphasis>cannot</emphasis> rely on totally accurate
2252 instruction-level profiling. However, for almost all circumstances the data
2253 can be useful. Ideally a utility such as Intel's VTUNE would be available to
2254 allow careful instruction-level analysis; go hassle Intel for this, not me ;)
2256 <sect1 id="irq-latency">
2257 <title>Profiling interrupt latency</title>
2259 This is an example of how the latency of delivery of profiling interrupts
2260 can impact the reliability of the profiling data. This is pretty much a
2261 worst-case-scenario example: these problems are fairly rare.
2264 double fun(double a, double b, double c)
2267 for (int i = 0 ; i < 10000; ++i) {
2276 Here the last instruction of the loop is very costly, and you would expect the result
2277 reflecting that - but (cutting the instructions inside the loop):
2280 $ opannotate -a -t 10 ./a.out
2282 88 15.38% : 8048337: fadd %st(3),%st
2283 48 8.391% : 8048339: fmul %st(2),%st
2284 68 11.88% : 804833b: fdiv %st(1),%st
2285 368 64.33% : 804833d: inc %eax
2286 : 804833e: cmp $0x270f,%eax
2287 : 8048343: jle 8048337
2290 The problem comes from the x86 hardware; when the counter overflows the IRQ
2291 is asserted but the hardware has features that can delay the NMI interrupt:
2292 x86 hardware is synchronous (i.e. cannot interrupt during an instruction);
2293 there is also a latency when the IRQ is asserted, and the multiple
2294 execution units and the out-of-order model of modern x86 CPUs also causes
2295 problems. This is the same function, with annotation :
2298 $ opannotate -s -t 10 ./a.out
2300 :double fun(double a, double b, double c)
2301 :{ /* _Z3funddd total: 572 100.0% */
2302 : double result = 0;
2303 368 64.33% : for (int i = 0 ; i < 10000; ++i) {
2304 88 15.38% : result += a;
2305 48 8.391% : result *= b;
2306 68 11.88% : result /= c;
2312 The conclusion: don't trust samples coming at the end of a loop,
2313 particularly if the last instruction generated by the compiler is costly. This
2314 case can also occur for branches. Always bear in mind that samples
2315 can be delayed by a few cycles from its real position. That's a hardware
2316 problem and OProfile can do nothing about it.
2319 <sect1 id="kernel-profiling">
2320 <title>Kernel profiling</title>
2321 <sect2 id="irq-masking">
2322 <title>Interrupt masking</title>
2324 OProfile uses non-maskable interrupts (NMI) on the P6 generation, Pentium 4,
2325 Athlon, Opteron, Phenom, and Turion processors. These interrupts can occur even in sections of the
2326 kernel where interrupts are disabled, allowing collection of samples in virtually
2327 all executable code.
2331 <title>Idle time</title>
2333 Your kernel is likely to support halting the processor when a CPU is idle. As
2334 the typical hardware events like <constant>CPU_CLK_UNHALTED</constant> do not
2335 count when the CPU is halted, the kernel profile will not reflect the actual
2336 amount of time spent idle. You can change this behaviour by booting with
2337 the <option>idle=poll</option> option, which uses a different idle routine. This
2338 will appear as <function>poll_idle()</function> in your kernel profile.
2341 <sect2 id="kernel-modules">
2342 <title>Profiling kernel modules</title>
2344 OProfile profiles kernel modules by default. However, there are a couple of problems
2345 you may have when trying to get results. First, you may have booted via an initrd;
2346 this means that the actual path for the module binaries cannot be determined automatically.
2347 To get around this, you can use the <option>-p</option> option to the analysis tools
2348 to specify where to look for the kernel modules.
2351 In kernel version 2.6, the information on where kernel module binaries are located was removed.
2352 This means OProfile needs guiding with the <option>-p</option> option to find your
2353 modules. Normally, you can just use your standard module top-level directory for this.
2354 Note that due to this problem, OProfile cannot check that the modification times match;
2355 it is your responsibility to make sure you do not modify a binary after a profile
2359 If you have run <command>insmod</command> or <command>modprobe</command> to insert a module
2360 in a particular directory, it is important that you specify this directory with the
2361 <option>-p</option> option first, so that it over-rides an older module binary that might
2362 exist in other directories you've specified with <option>-p</option>. It is up to you
2363 to make sure that these values are correct: the kernel simply does not provide enough
2364 information for OProfile to get this information.
2369 <sect1 id="interpreting-callgraph">
2370 <title>Interpreting call-graph profiles</title>
2372 Sometimes the results from call-graph profiles may be different from what
2373 you expect to see. The first thing to check is whether the target
2374 binaries where compiled with frame pointers enabled (if the binary was
2375 compiled using <command>gcc</command>'s
2376 <option>-fomit-frame-pointer</option> option, you will not get
2377 meaningful results). Note that as of this writing, the GCC developers
2378 plan to disable frame pointers by default. The Linux kernel is built
2379 without frame pointers by default; there is a configuration option you
2380 can use to turn it on under the "Kernel Hacking" menu.
2383 Often you may see a caller of a function that does not actually directly
2384 call the function you're looking at (e.g. if <function>a()</function>
2385 calls <function>b()</function>, which in turn calls
2386 <function>c()</function>, you may see an entry for
2387 <function>a()->c()</function>). What's actually occurring is that we
2388 are taking samples at the very start (or the very end) of
2389 <function>c()</function>; at these few instructions, we haven't yet
2390 created the new function's frame, so it appears as if
2391 <function>a()</function> is calling directly into
2392 <function>c()</function>. Be careful not to be misled by these
2396 Like the rest of OProfile, call-graph profiling uses a statistical
2397 approach; this means that sometimes a backtrace sample is truncated, or
2398 even partially wrong. Bear this in mind when examining results.
2400 <!-- FIXME: what do we need here ? -->
2403 <sect1 id="debug-info">
2404 <title>Inaccuracies in annotated source</title>
2405 <sect2 id="effect-of-optimizations">
2406 <title>Side effects of optimizations</title>
2408 The compiler can introduce some pitfalls in the annotated source output.
2409 The optimizer can move pieces of code in such manner that two line of codes
2410 are interlaced (instruction scheduling). Also debug info generated by the compiler
2411 can show strange behavior. This is especially true for complex expressions e.g. inside
2420 here the problem come from the position of line number. The available debug
2421 info does not give enough details for the if condition, so all samples are
2422 accumulated at the position of the right brace of the expression. Using
2423 <command>opannotate <option>-a</option></command> can help to show the real
2424 samples at an assembly level.
2427 <sect2 id="prologues">
2428 <title>Prologues and epilogues</title>
2430 The compiler generally needs to generate "glue" code across function calls, dependent
2431 on the particular function call conventions used. Additionally other things
2432 need to happen, like stack pointer adjustment for the local variables; this
2433 code is known as the function prologue. Similar code is needed at function return,
2434 and is known as the function epilogue. This will show up in annotations as
2435 samples at the very start and end of a function, where there is no apparent
2436 executable code in the source.
2439 <sect2 id="inlined-function">
2440 <title>Inlined functions</title>
2442 You may see that a function is credited with a certain number of samples, but
2443 the listing does not add up to the correct total. To pick a real example :
2446 :internal_sk_buff_alloc_security(struct sk_buff *skb)
2447 353 2.342% :{ /* internal_sk_buff_alloc_security total: 1882 12.48% */
2449 : sk_buff_security_t *sksec;
2450 15 0.0995% : int rc = 0;
2452 10 0.06633% : sksec = skb->lsm_security;
2453 468 3.104% : if (sksec && sksec->magic == DSI_MAGIC) {
2457 : sksec = (sk_buff_security_t *) get_sk_buff_memory(skb);
2458 3 0.0199% : if (!sksec) {
2459 38 0.2521% : rc = -ENOMEM;
2462 : memset(sksec, 0, sizeof (sk_buff_security_t));
2463 44 0.2919% : sksec->magic = DSI_MAGIC;
2464 32 0.2123% : sksec->skb = skb;
2465 45 0.2985% : sksec->sid = DSI_SID_NORMAL;
2466 31 0.2056% : skb->lsm_security = sksec;
2470 146 0.9685% : return rc;
2475 Here, the function is credited with 1,882 samples, but the annotations
2476 below do not account for this. This is usually because of inline functions -
2477 the compiler marks such code with debug entries for the inline function
2478 definition, and this is where <command>opannotate</command> annotates
2479 such samples. In the case above, <function>memset</function> is the most
2480 likely candidate for this problem. Examining the mixed source/assembly
2481 output can help identify such results.
2484 This problem is more visible when there is no source file available, in the
2485 following example it's trivially visible the sums of symbols samples is less
2486 than the number of the samples for this file. The difference must be accounted
2487 to inline functions.
2491 * Total samples for file : "arch/i386/kernel/process.c"
2496 /* default_idle total: 84 1.8970 */
2497 /* cpu_idle total: 21 0.4743 */
2498 /* flush_thread total: 1 0.0226 */
2499 /* prepare_to_copy total: 1 0.0226 */
2500 /* __switch_to total: 18 0.4065 */
2503 The missing samples are not lost, they will be credited to another source
2504 location where the inlined function is defined. The inlined function will be
2505 credited from multiple call site and merged in one place in the annotated
2506 source file so there is no way to see from what call site are coming the
2507 samples for an inlined function.
2510 When running <command>opannotate</command>, you may get a warning
2511 "some functions compiled without debug information may have incorrect source line attributions".
2512 In some rare cases, OProfile is not able to verify that the derived source line
2513 is correct (when some parts of the binary image are compiled without debugging
2514 information). Be wary of results if this warning appears.
2517 Furthermore, for some languages the compiler can implicitly generate functions,
2518 such as default copy constructors. Such functions are labelled by the compiler
2519 as having a line number of 0, which means the source annotation can be confusing.
2521 <!-- FIXME so what *actually* happens to those samples ? ignored ? -->
2523 <sect2 id="wrong-linenr-info">
2524 <title>Inaccuracy in line number information</title>
2526 Depending on your compiler you can fall into the following problem:
2529 struct big_object { int a[500]; };
2534 for (int i = 0 ; i != 1000 * 1000; ++i)
2541 Compiled with <command>gcc</command> 3.0.4 the annotated source is clearly inaccurate:
2545 :{ /* main total: 7871 100% */
2547 : for (int i = 0 ; i != 1000 * 1000; ++i)
2549 7871 100% : return 0;
2553 The problem here is distinct from the IRQ latency problem; the debug line number
2554 information is not precise enough; again, looking at output of <command>opannoatate -as</command> can help.
2560 : for (int i = 0 ; i != 1000 * 1000; ++i)
2561 : 80484c0: push %ebp
2562 : 80484c1: mov %esp,%ebp
2563 : 80484c3: sub $0xfac,%esp
2564 : 80484c9: push %edi
2565 : 80484ca: push %esi
2566 : 80484cb: push %ebx
2568 : 80484cc: lea 0xfffff060(%ebp),%edx
2569 : 80484d2: lea 0xfffff830(%ebp),%eax
2570 : 80484d8: mov $0xf423f,%ebx
2571 : 80484dd: lea 0x0(%esi),%esi
2573 3 0.03811% : 80484e0: mov %edx,%edi
2574 : 80484e2: mov %eax,%esi
2575 1 0.0127% : 80484e4: cld
2576 8 0.1016% : 80484e5: mov $0x1f4,%ecx
2577 7850 99.73% : 80484ea: repz movsl %ds:(%esi),%es:(%edi)
2578 9 0.1143% : 80484ec: dec %ebx
2579 : 80484ed: jns 80484e0
2580 : 80484ef: xor %eax,%eax
2588 So here it's clear that copying is correctly credited with of all the samples, but the
2589 line number information is misplaced. <command>objdump -dS</command> exposes the
2590 same problem. Note that maintaining accurate debug information for compilers when optimizing is difficult, so this problem is not suprising.
2591 The problem of debug information
2592 accuracy is also dependent on the binutils version used; some BFD library versions
2593 contain a work-around for known problems of <command>gcc</command>, some others do not. This is unfortunate but we must live with that,
2594 since profiling is pointless when you disable optimisation (which would give better debugging entries).
2598 <sect1 id="symbol-without-debug-info">
2599 <title>Assembly functions</title>
2601 Often the assembler cannot generate debug information automatically.
2602 This means that you cannot get a source report unless
2603 you manually define the neccessary debug information; read your assembler documentation for how you might
2605 debugging info needed currently by OProfile is the line-number/filename-VMA association. When profiling assembly
2606 without debugging info you can always get report for symbols, and optionally for VMA, through <command>opreport -l</command>
2607 or <command>opreport -d</command>, but this works only for symbols with the right attributes.
2608 For <command>gas</command> you can get this by
2615 whilst for <command>nasm</command> you must use
2618 GLOBAL foo:function ; [1]
2621 Note that OProfile does not need the global attribute, only the function attribute.
2626 FIXME: I commented this bit out until we've written something ...
2628 improve this ? but look first why this file is special
2629 <sect2 id="small-functions">
2630 <title>Small functions</title>
2632 Very small functions can show strange behavior. The file in your source
2633 directory of OProfile <filename>$SRC/test-oprofile/understanding/puzzle.c</filename>
2639 <sect1 id="overlapping-symbols">
2640 <title>Overlapping symbols in JITed code</title>
2642 Some virtual machines (e.g., Java) may re-JIT a method, resulting in previously
2643 allocated space for a piece of compiled code to be reused. This means that, at one distinct
2644 code address, multiple symbols/methods may be present during the run time of the application.
2647 Since OProfile samples are buffered and don′t have timing information, there is no way
2648 to correlate samples with the (possibly) varying address ranges in which the code for a symbol
2650 An alternative would be flushing the OProfile sampling buffer when we get an unload event,
2651 but this could result in high overhead.
2654 To moderate the problem of overlapping symbols, OProfile tries to select the symbol that was
2655 present at this address range most of the time. Additionally, other overlapping symbols
2656 are truncated in the overlapping area.
2657 This gives reasonable results, because in reality, address reuse typically takes place
2658 during phase changes of the application -- in particular, during application startup.
2659 Thus, for optimum profiling results, start the sampling session after application startup
2664 <sect1 id="interpreting_operf_results">
2665 <title>Using operf to profile fork/execs</title>
2667 When profiling an application that forks one or more new processes, <command>operf</command> will
2668 record samples for both the parent process and forked processes. This is also true even if the
2669 forked process performs an exec of some sort (e.g., <code>execvp</code>). If the
2670 process does <emphasis>not</emphasis> perform an exec, you will see that <command>opreport</command>
2671 will attribute samples for the forked process to the main application executable. On the other
2672 hand, if the forked process <emphasis>does</emphasis> perform an exec, then <command>opreport</command>
2673 will attribute samples to the executable being exec'ed.
2676 To demonstrate this, consider the following examples.
2677 When using <command>operf</command> to profile a single application (either with the <code>--pid</code>
2678 option or <code>command</code> option), the normal <command>opreport</command> summary output
2679 (i.e., invoking <command>opreport</command> with no options) looks something like the following:
2684 112342 100.000 sprintft
2688 104209 92.7605 libc-2.12.so
2689 7273 6.4740 sprintft
2690 858 0.7637 no-vmlinux
2695 But if you profile an application that does a fork/exec, the <command>opreport</command> summary output
2696 will show samples for both the main application you profiled, as well as the exec'ed program.
2697 An example is shown below where <code>s-m-fork</code> is the main application being profiled, which
2698 in turn forks a process that does an <code>execvp</code> of the <code>memcpyt</code> program.
2703 133382 70.5031 memcpyt
2707 123852 92.8551 libc-2.12.so
2709 1007 0.7550 no-vmlinux
2710 1 7.5e-04 ld-2.12.so
2711 55804 29.4969 s-m-fork
2715 51801 92.8267 libc-2.12.so
2716 3589 6.4314 s-m-fork
2717 414 0.7419 no-vmlinux
2722 <sect1 id="hidden-cost">
2723 <title>Other discrepancies</title>
2725 Another cause of apparent problems is the hidden cost of instructions. A very
2726 common example is two memory reads: one from L1 cache and the other from memory:
2727 the second memory read is likely to have more samples.
2728 There are many other causes of hidden cost of instructions. A non-exhaustive
2729 list: mis-predicted branch, TLB cache miss, partial register stall,
2730 partial register dependencies, memory mismatch stall, re-executed µops. If you want to write
2731 programs at the assembly level, be sure to take a look at the Intel and
2732 AMD documentation at <ulink url="http://developer.intel.com/">http://developer.intel.com/</ulink>
2733 and <ulink url="http://developer.amd.com/devguides.jsp/">http://developer.amd.com/devguides.jsp</ulink>.
2738 <chapter id="controlling-counter">
2739 <title>Controlling the event counter</title>
2740 <sect1 id="controlling-ocount">
2741 <title>Using <command>ocount</command></title>
2743 This section describes in detail how <command>ocount</command> is used.
2744 Unless the <option>--events</option> option is specified, <command>ocount</command> will use
2745 the default event for your system. For most systems, the default event is some
2746 cycles-based event, assuming your processor type supports hardware performance
2747 counters. The event specification used for <command>ocount</command> is slightly
2748 different from that required for profiling -- a <emphasis>count</emphasis> value
2749 is not needed. You can see the event information for your CPU using <command>ophelp</command>.
2750 More information on event specification can be found at <xref linkend="eventspec"/>.
2753 The <command>ocount</command> command syntax is:
2755 <screen>ocount [ options ] [ --system-wide | --process-list <pids> | --thread-list <tids> | --cpu-list <cpus> [ command [ args ] ] ]
2759 <command>ocount</command> has 5 run modes:
2762 <listitem>system-wide</listitem>
2763 <listitem>process-list</listitem>
2764 <listitem>thread-list</listitem>
2765 <listitem>cpu-list</listitem>
2766 <listitem>command</listitem>
2770 One and only one of these 5 run modes must be specified when you run <command>ocount</command>.
2771 If you run <command>ocount</command> using a run mode other than <code>command [args]</code>, press Ctrl-c
2772 to stop it when finished counting (e.g., when the monitored process ends). If you background <command>ocount</command>
2773 (i.e., with ’&’) while using one these run modes, you must stop it in a controlled manner so that
2774 the data collection process can be shut down cleanly and final results can be displayed.
2775 Use <code>kill -SIGINT <ocount-PID></code> for this purpose.
2778 Following is a description of the <command>ocount</command> options.
2782 <term><option>command [args]</option></term>
2784 The command or application to be profiled. The <emphasis>[args]</emphasis> are the input arguments
2785 that the command or application requires. The command and its arguments must be positioned at the
2786 end of the command line, after all other <command>ocount</command> options.
2790 <term><option>--process-list / -p [PIDs]</option></term>
2792 Use this option to count events for one or more already-running applications, specified via
2793 a comma-separated list (PIDs). Event counts will be collected for all children of the
2794 passed process(es) as well.
2798 <term><option>--thread-list / -r [TIDs]</option></term>
2800 Use this option to count events for one or more already-running threads, specified via
2801 a comma-separated list (TIDs). Event counts will <emphasis>not</emphasis> be collected
2802 for any children of the passed thread(s).
2806 <term><option>--system-wide / -s</option></term>
2808 This option is for counting events for all processes running on your system. You must have
2809 root authority to run <command>ocount</command> in this mode.
2813 <term><option>--cpu-list / -C [CPUs]</option></term>
2815 This option is for counting events on a subset of processors on your system. You must have
2816 root authority to run <command>ocount</command> in this mode. This is a comma-separated list,
2817 where each element in the list may be either a single processor number or a range of processor
2818 numbers; for example: ’-C 2,3,4-11,15’.
2822 <term><option>--events / -e [event1[,event2[,...]]]</option></term>
2824 This option is for passing a comma-separated list of event specifications
2825 for counting. Each event spec is of the form:
2827 <screen>name[:unitmask[:kernel[:user]]]</screen>
2829 When no event specification is given, the default event for the running
2830 processor type will be used for counting. Use <command>ophelp</command>
2831 to list the available events for your processor type.
2835 <term><option>--separate-thread / -t</option></term>
2837 This option can be used in conjunction with either the <code>--process-list</code> or
2838 <code>--thread-list</code> option to display event counts on a per-thread (per-process) basis.
2839 Without this option, all counts are aggregated.
2843 <term><option>--separate-cpu / -c</option></term>
2845 This option can be used in conjunction with either the <code>--system-wide</code> or
2846 <code>--cpu-list</code> option to display event counts on a per-cpu basis. Without this option,
2847 all counts are aggregated.
2851 <term><option>--time-interval / -i interval_length[:num_intervals]</option></term>
2853 <command>Note: </command>The <code>interval_length</code> is given in milliseconds.
2854 However, the current implementation only supports 100 ms
2855 granularity, so the given <code>interval_length</code> will be rounded
2856 to the nearest 100 ms. Results collected for each time
2857 interval are printed immediately instead of the default
2858 of one dump of cumulative event counts at the end of the
2859 run. Counters are reset to zero at the start of each
2863 If <code>num_intervals</code> is specified, ocount exits after the
2864 specified number of intervals occur.
2868 <term><option>--brief-format / -b</option></term>
2870 Use this option to print results in the following brief format:
2872 [optional cpu or thread,]<event_name>,<count>,<percent_time_enabled>
2873 [ <int> ,]< string >,< u64 >,< double >
2875 If <code>--timer-interval</code> is specified, a separate line formatted as
2877 timestamp,<num_seconds_since_epoch>[.n]
2879 is printed ahead of each dump of event counts. If the time interval specified is
2880 less than one second, the timestamp will have 1/10 second precision.
2885 <term><option>--output-file / -f outfile_name</option></term>
2887 Results are written to outfile_name instead of interactively to the terminal.
2891 <term><option>--verbose / -V</option></term>
2893 Use this option to increase the verbosity of the output.
2897 <term><option>--version -v </option></term>
2899 Show <command>ocount</command> version.
2903 <term><option>--help / -h</option></term>
2905 Show a help message.
2915 <title>Acknowledgments</title>
2917 Thanks to (in no particular order) : Arjan van de Ven, Rik van Riel, Juan Quintela, Philippe Elie,
2918 Phillipp Rumpf, Tigran Aivazian, Alex Brown, Alisdair Rawsthorne, Bob Montgomery, Ray Bryant, H.J. Lu,
2919 Jeff Esper, Will Cohen, Graydon Hoare, Cliff Woolley, Alex Tsariounov, Al Stone, Jason Yeh,
2920 Randolph Chung, Anton Blanchard, Richard Henderson, Andries Brouwer, Bryan Rittmeyer,
2922 Richard Reich (rreich@rdrtech.com), Zwane Mwaikambo, Dave Jones, Charles Filtness; and finally Pulp, for "Intro".
projects / platform / upstream / oprofile.git / blob