3 This document describes the GStreamer caps negotiation of DMA buffers on
6 The DMA buffer sharing is the efficient way to share the buffer/memory
7 between different Linux kernel driver, such as codecs/3D/display/cameras.
8 For example, the decoder may want its output to be directly shared with the
9 display server for rendering without a copy.
11 Any device driver which is part of DMA buffer sharing, can do so as either
12 the *exporter* or *importer* of buffers.
14 This kind of buffer/memory is usually stored in non-system memory (maybe in
15 device's local memory or something else not directly accessible by the
16 CPU), then its memory mapping for CPU access may impose a big overhead and
17 low performance, or even impossible.
19 DMA buffers are exposed to user-space as *file descriptors* allowing to pass
20 them between processes.
25 PRIME is the cross device buffer sharing framework in DRM kernel
26 subsystem. These are the ones normally used in GStreamer which might
29 PRIME buffers requires some metadata to describe how to interpret them,
30 such as a set of file descriptors (for example, one per plane), color
31 definition in fourcc, and DRM-modifiers. If the frame is going to be mapped
32 onto system's memory, also is needed padding, strides, offsets, etc.
37 Each file descriptor represents a chunk of a frame, usually a plane. For
38 example, when a DMA buffer contains NV12 format data, it might be
39 composited by 2 planes: one for its Y component and the other for both UV
40 components. Then, the hardware may use two detached memory chunks, one per
41 plane, exposed as two file descriptors. Otherwise, if hardware uses only
42 one continuous memory chunk for all the planes, the DMA buffer should just
43 have one file descriptor.
48 Just like fourcc common usage, DRM-fourcc describes the underlying format
49 of the video frame, such as `DRM_FORMAT_YVU420` or `DRM_FORMAT_NV12`. All
50 of them with the prefix `DRM_FORMAT_`. Please refer to `drm_fourcc.h` in
51 the kernel for a full list. This list of fourcc formats maps to GStreamer
52 video formats, although the GStreamer formats may have a slighly different.
53 For example, DRM_FORMAT_ARGB8888 corresponds to GST_VIDEO_FORMAT_BGRA.
58 DRM-modifier describes the translation mechanism between pixel to memory
59 samples and the actual memory storage of the buffer. The most
60 straightforward modifier is LINEAR, where each pixel has contiguous storage
61 and pixel location in memory can be easily calculated with the stride. This
62 is considered the baseline interchange format, and most convenient for CPU
63 access. Nonetheless, modern hardware employs more sophisticated memory
64 access mechanisms, such as tiling and possibly compression. For example,
65 the TILED modifier describes memory storage where pixels are stored in 4x4
66 blocks arranged in row-major ordering. For example, the first tile in
67 memory stores pixels (0,0) to (3,3) inclusive, and the second tile in
68 memory stores pixels (4,0) to (7,3) inclusive, and so on.
70 DRM-modifier is a sixteen hexadecimal digits to represent these memory
71 layouts. For example, `0x0000000000000000` means linear,
72 `0x0100000000000001` means Intel's X tile mode, etc. Please refer to
73 `drm_fourcc.h` in kernel for a full list.
75 Excepting the linear modifier, the first 8 bits represent the vendor ID and
76 the other 56 bits describe the memory layout, which may be hardware
77 dependent. Users should be careful when interpreting non-linear memory by
80 Please bear in mind that, even for the linear modifier, as the access to
81 DMA memory's content is through `map()` / `unmap()` functions, its
82 read/write performance may be low or even bad, because of its cache type
83 and coherence assurance. So, most of the times, it's advised to avoid that
84 code path for upload or download frame data.
89 The meta data contains information about how to interpret the memory
90 holding the video frame, either when the frame mapped and its DRM modifier
91 is linear, or by other API that imports those DMA buffers.
94 # DMABufs in GStreamer
99 In GStreamer, a full DMA buffer-based video frame is mapped to a
100 `GstBuffer`, and each file descriptor used to describe the whole frame is
101 held by a `GstMemory` mini-object. A derived class of `GstDmaBufAllocator`
102 would be implemented for every wrapped API *exporting* DMA buffers to
103 user-space, as memory allocator.
106 ## DRM format caps field
108 The *GstCapsFeatures* *memory:DMABuf* is usually used to negotiate DMA
109 buffers. It is recommended to allow DMAbuf to flow without the
110 *GstCapsFeatures* *memory:DMABuf* if the DRM-modifier is linear.
112 But also, in order to negotiate *memory:DMABuf* thoroughly, it's required
113 to match the DRM-modifiers between upstream and downstream. Otherwise video
114 sinks might end rendering wrong frames assuming linear access.
116 Because DRM-fourcc and DRM-modifier are both necessary to render frames
117 DMABuf-backed, we now consider both as a pair and combine them together to
118 assure uniqueness. In caps, we use a *:* to link them together and write in
119 the mode of *DRM_FORMAT:DRM_MODIFIER*, which represents a totally new single video
120 format. For example, `NV12:0x0100000000000002` is a new video format
121 combined by video format NV12 and the modifier `0x0100000000000002`. It's
122 not NV12 and it's not its subset either.
124 *DRM_FORMAT* can be printed by using
125 `GST_FOURCC_FORMAT` and `GST_FOURCC_ARGS` macros from the
126 `DRM_FORMAT_*` constants, it is NOT a `GstVideoFormat`, so it would be
127 different from the content of the `format` field in a non-dmabuf caps.
128 A modifier must always be present, except if the modifier is linear,
129 then it should not be included, so `NV12:0x0000000000000000` is
130 invalid, it must be `drm-format=NV12`. DRM fourcc are used
131 instead of a `GstVideoFormat` to make it easier for non-GStreamer
132 developers to understand what the system is trying to achieve.
134 Please note that this form of video format only appears within
135 *memory:DMABuf* feature. It must not appear in any other video caps
138 Unlike other type of video buffers, DMABuf frames might not be mappable and
139 its internal format is opaque to the user. Then, unless the modifier is
140 linear (0x0000000000000000) or some other well known tiled format such as
141 NV12_4L4, NV12_16L16, NV12_64Z32, NV12_16L32S, etc. (which are defined in
142 video-format.h), we always use `GST_VIDEO_FORMAT_ENCODED` in
143 `GstVideoFormat` enum to represent its video format.
145 In order to not misuse this new format with the common video format, **in**
146 *memory:DMABuf* feature, *drm-format* field in caps will replace the
147 traditional *format* field.
149 So a DMABuf-backed video caps may look like:
152 video/x-raw(memory:DMABuf), \
153 drm-format=(string)NV12:0x0x0100000000000001, \
156 interlace-mode=(string)progressive, \
157 multiview-mode=(string)mono, \
158 multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, \
159 pixel-aspect-ratio=(fraction)1/1, \
160 framerate=(fraction)24/1, \
161 colorimetry=(string)bt709"
164 And when we call a video info API such as `gst_video_info_from_caps()` with
165 this caps, it should return an video format as `GST_VIDEO_FORMAT_ENCODED`,
166 leaving other fields unchanged as normal video caps.
168 In addition, a new structure
171 struct GstDrmVideoInfo
175 guint64 drm_modifier;
179 is introduced to represent more info of DMA video caps. User should use
180 this DMABuf related API such as `gst_drm_video_info_from_caps()` to recognize
181 the video format and parse the DMA info from caps.
186 Besides the *file descriptors*, there may be a `GstVideoMeta` data attached
187 to each `GstBuffer` to describe more information such as the width, height,
188 pitches, strides and plane offsets for that DMA buffer (Please note that
189 the mandatory width and height information appears both in "caps" and here,
190 and they should be always equal). This kind of information is only obtained
191 by each module's API, such as the functions
192 `VkImageDrmFormatModifierExplicitCreateInfoEXT()` in Vulkan, and
193 `vaExportSurfaceHandle()` in VA-API. The information should be translated
194 into `GstVideoMeta`'s fields when the DMA buffer is created and
195 exported. These meta data is useful when other module wants to import the
198 For example, we may create a `GstBuffer` using `vaExportSurfaceHandle()`
199 VA-API, and set each field of `GstVideoMeta` with information from
200 `VADRMPRIMESurfaceDescriptor`. Later, a downstream Vulkan element imports
201 these DMA buffers with `VkImageDrmFormatModifierExplicitCreateInfoEXT()`,
202 translating fields form buffer's `GstVideoMeta` into the
203 `VkSubresourceLayout` parameter.
205 In short, the `GstVideoMeta` contains the common extra video information
206 about the DMA buffer, which can be interpreted by each module.
208 Information in `GstVideoMeta` depends on the hardware context and
209 setting. Its values, such as stride and pitch, may differ from the standard
210 video format because of the hardware's requirement. For example, if a DMA
211 buffer represents a compressed video in memory, its pitch and stride may be
212 smaller than the standard linear one because of the compression. Please
213 remind that users should not use this meta data to interpret and access the
214 DMA buffer, **unless the modifier is linear**.
217 # Negotiation of DMA buffer
219 If two elements of different modules (for example, VA-API decoder to
220 Wayland sink) want to transfer dmabufs, the negotiation should ensure a
221 common *drm-format* (*DRM_FORMAT:DRM_MODIFIER*). As we already illustrate how to
222 represent both of them in caps before, so the negotiation here in fact has
223 no special operation except finding the intersection.
226 ## Static Template Caps
228 If an element can list all the DRM fourcc/modifier composition at register
229 time, `gst-inspect` result should look like:
235 video/x-raw(memory:DMABuf)
237 height: [ 16, 16384 ]
238 drm-format: { (string)NV12:0x0100000000000001, \
239 (string)YU12, (string)YV12, \
240 (string)YUYV:0x0100000000000002, \
241 (string)P010:0x0100000000000002, \
242 (string)AR24:0x0100000000000002, \
243 (string)AB24:0x0100000000000002, \
244 (string)AR39:0x0100000000000002, \
245 (string)AYUV:0x0100000000000002 }
248 But because sometimes it is impossible to enumerate and list all
249 drm_fourcc/modifier composition in static templates (for example, we may
250 need a runtime context which is not available at register time to detect
251 the real modifers a HW can support), we can let the *drm-format* field
252 absent to mean the super set of all formats.
257 Sometimes, a renegotiation may happen if the downstream element is not
258 pleased with the caps set by the upstream element. For example, some sink
259 element may not know the preferred DRM fourcc/modifier until the real
260 render target window is realized. Then, it will send a "reconfigure" event
261 to upstream element to require a renegotiation. At this round negotiation,
262 the downstream element will provide a more precise *drm-format* list.
267 Consider the pipeline of:
270 vapostproc ! video/x-raw(memory:DMABuf) ! glupload
273 both `vapostproc` and `glupload` work on the same GPU. (DMABuf caps filter
274 is just for illustration, it doesn't need to be specified, since DMA
275 negotiation is well supported.)
277 The VA-API based `vapostproc` element can detect the modifiers at the
278 element registration time and the src template should be:
284 video/x-raw(memory:DMABuf)
286 height: [ 16, 16384 ]
287 drm-format: { (string)NV12:0x0100000000000001, \
288 (string)NV12, (string)I420, (string)YV12, \
289 (string)BGRA:0x0100000000000002 }
292 While `glupload` needs the runtime EGL context to check the DRM fourcc and
293 modifiers, so it can just leave the *drm-format* field absent in its sink
297 SINK template: 'sink'
300 video/x-raw(memory:DMABuf)
301 width: [ 1, 2147483647 ]
302 height: [ 1, 2147483647 ]
305 At runtime, when the `vapostproc` wants to decide its src caps, it first
306 query the downstream `glupload` element about all possible DMA caps. The
307 `glupload` should answer that query based on the GL/EGL query result, such
311 drm-format: { (string)NV12:0x0100000000000001, (string)BGRA }
314 So, the intersection with `vapostproc`'s src caps will be
315 `NV12:0x0100000000000001`. It will be the sent to downstream (`glupload`)
316 by a CAPS event. The `vapostproc` element may also query the allocation
317 after that CAPS event, but downstream `glupload` will not provide a DMA
318 buffer pool because EGL API is mostly for DMAbuf importing. Then
319 `vapostproc` will create its own DMA pool, the buffers created from that
320 new pool should conform *drm-format*, described in this document, with
321 `NV12:0x0100000000000001`. Also, the downstream `glupload` should make sure
322 that it can import other DMA buffers which are not created in the pool it
323 provided, as long as they conform with *drm-format*
324 `NV12:0x0100000000000001`.
326 Then, when `vapostproc` handles each frame, it creates GPU surfaces with
327 *drm-format* `NV12:0x0100000000000001`. Each surface is also exported as a
328 set of file descriptors, each one wrapped in `GstMemory` allocated by a
329 subclass of `GstDmaBufAllocator`. All the `GstMemory` are appended to a
330 `GstBuffer`. There may be some extra information about the pitch, stride
331 and plane offset when we export the surface, we also need to translate them
332 into `GstVideoMeta` and attached it to the `GstBuffer`.
334 Later `glupload`, when it receives a `GstBuffer`, it can use those file
335 descriptors with *drm-format* `NV12:0x0100000000000001` to import an
336 EGLImage. If the `GstVideoMeta` exists, this extra parameters should also
337 be provided to the importing API.