4 Mesa has experimental support for `Perfetto <https://perfetto.dev>`__ for
5 GPU performance monitoring. Perfetto supports multiple
6 `producers <https://perfetto.dev/docs/concepts/service-model>`__ each with
7 one or more data-sources. Perfetto already provides various producers and
8 data-sources for things like:
10 - CPU scheduling events (``linux.ftrace``)
11 - CPU frequency scaling (``linux.ftrace``)
12 - System calls (``linux.ftrace``)
13 - Process memory utilization (``linux.process_stats``)
15 As well as various domain specific producers.
17 The mesa Perfetto support adds additional producers, to allow for visualizing
18 GPU performance (frequency, utilization, performance counters, etc) on the
19 same timeline, to better understand and tune/debug system level performance:
21 - pps-producer: A systemwide daemon that can collect global performance
23 - mesa: Per-process producer within mesa to capture render-stage traces
24 on the GPU timeline, track events on the CPU timeline, etc.
26 The exact supported features vary per driver:
28 .. list-table:: Supported data-sources
35 - ``gpu.counters.msm``
36 - ``gpu.renderstages.msm``
38 - ``gpu.counters.msm``
39 - ``gpu.renderstages.msm``
41 - ``gpu.counters.i915``
42 - ``gpu.renderstages.intel``
44 - ``gpu.counters.panfrost``
50 To capture a trace with Perfetto you need to take the following steps:
52 1. Build Perfetto from sources available at ``subprojects/perfetto`` following
53 `this guide <https://perfetto.dev/docs/quickstart/linux-tracing>`__.
55 2. Create a `trace config <https://perfetto.dev/docs/concepts/config>`__, which is
56 a json formatted text file with extension ``.cfg``, or use one of the config
57 files under the ``src/tool/pps/cfg`` directory. More examples of config files
58 can be found in ``subprojects/perfetto/test/configs``.
60 3. Change directory to ``subprojects/perfetto`` and run a
61 `convenience script <https://perfetto.dev/docs/quickstart/linux-tracing#capturing-a-trace>`__
62 to start the tracing service:
64 .. code-block:: console
66 cd subprojects/perfetto
67 CONFIG=<path/to/gpu.cfg> OUT=out/linux_clang_release ./tools/tmux -n
69 4. Start other producers you may need, e.g. ``pps-producer``.
71 5. Start ``perfetto`` under the tmux session initiated in step 3.
73 6. Once tracing has finished, you can detach from tmux with :kbd:`Ctrl+b`,
74 :kbd:`d`, and the convenience script should automatically copy the trace
75 files into ``$HOME/Downloads``.
77 7. Go to `ui.perfetto.dev <https://ui.perfetto.dev>`__ and upload
78 ``$HOME/Downloads/trace.protobuf`` by clicking on **Open trace file**.
80 8. Alternatively you can open the trace in `AGI <https://gpuinspector.dev/>`__
81 (which despite the name can be used to view non-android traces).
83 To be a bit more explicit, here is a listing of commands reproducing
86 .. code-block:: console
88 # Configure Mesa with perfetto
89 mesa $ meson . build -Dperfetto=true -Dvulkan-drivers=intel,broadcom -Dgallium-drivers=
93 # Within the Mesa repo, build perfetto
94 mesa $ cd subprojects/perfetto
95 perfetto $ ./tools/install-build-deps
96 perfetto $ ./tools/gn gen --args='is_debug=false' out/linux
97 perfetto $ ./tools/ninja -C out/linux
100 perfetto $ CONFIG=../../src/tool/pps/cfg/gpu.cfg OUT=out/linux/ ./tools/tmux -n
102 # In parallel from the Mesa repo, start the PPS producer
103 mesa $ ./build/src/tool/pps/pps-producer
105 # Back in the perfetto tmux, press enter to start the capture
110 Mesa's CPU tracepoints (``MESA_TRACE_*``) use Perfetto track events when
111 Perfetto is enabled. They use ``mesa.default`` and ``mesa.slow`` categories.
113 Currently, only EGL and Freedreno have CPU tracepoints.
118 The Vulkan API gives the application control over recording of command
119 buffers as well as when they are submitted to the hardware. As a
120 consequence, we need to ensure command buffers are properly
121 instrumented for the Perfetto driver data sources prior to Perfetto
122 actually collecting traces.
124 This can be achieved by setting the :envvar:`MESA_GPU_TRACES`
125 environment variable before starting a Vulkan application :
127 .. code-block:: console
129 MESA_GPU_TRACES=perfetto ./build/my_vulkan_app
134 Below is driver specific information/instructions for the PPS producer.
139 The Freedreno PPS driver needs root access to read system-wide
140 performance counters, so you can simply run it with sudo:
142 .. code-block:: console
144 sudo ./build/src/tool/pps/pps-producer
149 The Intel PPS driver needs root access to read system-wide
150 `RenderBasic <https://www.intel.com/content/www/us/en/develop/documentation/vtune-help/top/reference/gpu-metrics-reference.html>`__
151 performance counters, so you can simply run it with sudo:
153 .. code-block:: console
155 sudo ./build/src/tool/pps/pps-producer
157 Another option to enable access wide data without root permissions would be running the following:
159 .. code-block:: console
161 sudo sysctl dev.i915.perf_stream_paranoid=0
163 Alternatively using the ``CAP_PERFMON`` permission on the binary should work too.
165 A particular metric set can also be selected to capture a different
168 .. code-block:: console
170 INTEL_PERFETTO_METRIC_SET=RasterizerAndPixelBackend ./build/src/tool/pps/pps-producer
172 Vulkan applications can also be instrumented to be Perfetto producers.
173 To enable this for given application, set the environment variable as
176 .. code-block:: console
178 PERFETTO_TRACE=1 my_vulkan_app
183 The Panfrost PPS driver uses unstable ioctls that behave correctly on
184 kernel version `5.4.23+ <https://lwn.net/Articles/813601/>`__ and
185 `5.5.7+ <https://lwn.net/Articles/813600/>`__.
187 To run the producer, follow these two simple steps:
189 1. Enable Panfrost unstable ioctls via kernel parameter:
191 .. code-block:: console
193 modprobe panfrost unstable_ioctls=1
195 Alternatively you could add ``panfrost.unstable_ioctls=1`` to your kernel command line, or ``echo 1 > /sys/module/panfrost/parameters/unstable_ioctls``.
199 .. code-block:: console
209 If the convenience script ``tools/tmux`` keeps copying artifacts to your
210 ``SSH_TARGET`` without starting the tmux session, make sure you have ``tmux``
211 installed in your system.
213 .. code-block:: console
217 Missing counter names
218 ~~~~~~~~~~~~~~~~~~~~~
220 If the trace viewer shows a list of counters with a description like
221 ``gpu_counter(#)`` instead of their proper names, maybe you had a data loss due
222 to the trace buffer being full and wrapped.
224 In order to prevent this loss of data you can tweak the trace config file in
227 - Increase the size of the buffer in use:
229 .. code-block:: javascript
233 fill_policy: RING_BUFFER,
236 - Periodically flush the trace buffer into the output file:
238 .. code-block:: javascript
240 write_into_file: true
241 file_write_period_ms: 250
244 - Discard new traces when the buffer fills:
246 .. code-block:: javascript
250 fill_policy: DISCARD,