4 This document describes the design for arbitrary per-buffer metadata.
6 Buffer metadata typically describes the lowlevel properties of the buffer
7 content. These properties are typically not negotiated with caps but they are
8 negotiated in the bufferpools.
10 Some examples of metadata:
14 - interlacing information
15 - video alignment, cropping, panning information
16 - extra container information such as granulepos, ...
17 - extra global buffer properties
24 * allocation, free, low fragmentation
25 * access to the metadata fields, preferably not much slower than directly
26 accessing a C structure field
27 - It must be extensible. Elements should be able to add new arbitrary metadata
28 without requiring much effort. Also new metadata fields should not break API
30 - It plays nice with subbuffers. When a subbuffer is created, the various
31 buffer metadata should be copied/updated correctly.
32 - We should be able to negotiate metadata between elements
39 Video data is sometimes allocated in non-contiguous planes for the Y and the UV
40 data. We need to be able to specify the data on a buffer using multiple
41 pointers in memory. We also need to be able to specify the stride for these
46 Some elements might need to store extra data for a buffer. This is typically
47 done when the resources are allocated from another subsystem such as OMX or
50 * Processing information
52 Pan and crop information can be added to the buffer data when the downstream
53 element can understand and use this metadata. An imagesink can, for example,
54 use the pan and cropping formation when it blits the image on the screen
61 A GstMeta is a structure as follows:
64 const GstMetaInfo *info; /* tag and info for the meta item */
67 The purpose of the this structure is to serve as a common header for all metadata
68 information that we can attach to a buffer. Specific metadata, such as timing metadata,
69 will have this structure as the first field. For example:
71 struct _GstMetaTiming {
72 GstMeta meta; /* common meta header */
74 GstClockTime dts; /* decoding timestamp */
75 GstClockTime pts; /* presentation timestamp */
76 GstClockTime duration; /* duration of the data */
77 GstClockTime clock_rate; /* clock rate for the above values */
80 Or another example for the video memory regions that consists of both fields and
84 #define GST_VIDEO_MAX_PLANES 4
86 struct GstMetaVideoPlane {
87 gsize offset; /* offset in the buffer memory region of the
89 gint stride; /* stride of the image lines. Can be negative when
90 * the image is upside-down */
96 GstMetaVideoFlags flags
99 GstVideoPlane plane[GST_VIDEO_MAX_PLANES];
101 gpointer (*map) (GstMetaVideo *meta, guint plane, gint *stride,
103 gboolean (*unmap) (GstMetaVideo *meta, guint plane, gpointer data);
106 gpointer gst_meta_video_map (GstMetaVideo *meta, guint plane, gint *stride,
108 gboolean gst_meta_video_unmap (GstMetaVideo *meta, guint plane, gpointer data);
110 GstMeta derived structures define the API of the metadata. The API can consist of
111 fields and/or methods. It is possible to have different implementations for the
112 same GstMeta structure.
114 The implementation of the GstMeta api would typically add more fields to the
115 public structure that allow it to implement the API.
117 GstMetaInfo will point to more information about the metadata and looks like this:
119 struct _GstMetaInfo {
120 GQuark api; /* api name */
121 GQuark impl; /* implementation name */
122 gsize size; /* size of the structure */
124 GstMetaInitFunction init_func;
125 GstMetaFreeFunction free_func;
126 GstMetaCopyFunction copy_func;
127 GstMetaTransformFunction transform_func;
128 GstMetaSerializeFunction serialize_func
129 GstMetaDeserializeFunction deserialize_func
132 api will contain a GQuark of the metadata api. A repository of registered MetaInfo
133 will be maintained by the core. We will register some common metadata structures
134 in core and some media specific info for audio/video/text in -base. Plugins can
135 register additional custom metadata.
137 For each implementation of api, there will thus be a unique GstMetaInfo. In the
138 case of metadata with a well defined API, the implementation specific init
139 function will setup the methods in the metadata structure.
141 Along with the metadata description we will have functions to initialize/free (and/or refcount)
142 a specific GstMeta instance. We also have the possibility to add a custom
143 transform function that can be used to modify the metadata when a transformation
146 We also add serialize and deserialize function for the metadata in case we need special
147 logic for reading and writing the metadata. This is needed for GDP payloading of the
150 The purpose of the separate MetaInfo is to not have to carry the free/init functions in
151 each buffer instance but to define them globally. We still want quick access to the info
152 so we need to make the buffer metadata point to the info.
154 Technically we could also specify the field and types in the MetaInfo and
155 provide a generic API to retrieve the metadata fields without the need for a
156 header file. We will not do this yet.
158 Allocation of the GstBuffer structure will result in the allocation of a memory region
159 of a customizable size (512 bytes). Only the first sizeof (GstBuffer) bytes of this
160 region will initially be used. The remaining bytes will be part of the free metadata
161 region of the buffer. Different implementations are possible and are invisible
164 The complete buffer with metadata could, for example, look as follows:
166 +-------------------------------------+
167 GstMiniObject | GType (GstBuffer) |
168 | refcount, flags, copy/disp/free |
169 +-------------------------------------+
170 GstBuffer | caps, parent, pool |
171 +.....................................+
173 +- | info ------> GstMetaInfo
174 GstMetaTiming | | | |
179 + . . . . . . . . . . . . . . . . . . + |
181 GstMetaVideo +- +- | info ------> GstMetaInfo
189 | | private fields | |
190 GstMetaVideoImpl | | ... | |
193 + . . . . . . . . . . . . . . . . . . + .
200 Buffers are created using the normal gst_buffer_new functions. The standard fields
201 are initialized as usual. A memory area that is bigger than the structure size
202 is allocated for the buffer metadata.
206 After creating a buffer, the application can set caps and add metadata
209 To add or retrieve metadata, a handle to a GstMetaInfo structure needs to be
210 obtained. This defines the implementation and API of the metadata. Usually, a
211 handle to this info structure can be obtained by calling a public _get_info()
212 method from a shared library (for shared metadata).
214 The following defines can usually be found in the shared .h file.
216 GstMetaInfo * gst_meta_timing_get_info();
217 #define GST_META_TIMING_INFO (gst_meta_timing_get_info())
219 Adding metadata to a buffer can be done with the gst_buffer_add_meta() call.
220 This function will create new metadata based on the implementation specified by
221 the GstMetaInfo. It is alos possible to pass a generic pointer to the add_meta()
222 function that can contain parameters to initialize the new metadata fields.
224 Retrieving the metadata on a buffer can be done with the
225 gst_buffer_meta_get() method. This function retrieves an existing metadata
226 conforming to the API specified in the given info. When no such metadata exists,
227 the function will return NULL.
229 GstMetaTiming *timing;
231 timing = gst_buffer_get_meta (buffer, GST_META_TIMING_INFO);
233 Once a reference to the info has been obtained, the associated metadata can be
234 added or modified on a buffer.
236 timing->timestamp = 0;
237 timing->duration = 20 * GST_MSECOND;
239 Other convenience macros can be made to simplify the above code:
241 #define gst_buffer_get_meta_timing(b) \
242 ((GstMetaTiming *) gst_buffer_get_meta ((b), GST_META_TIMING_INFO)
244 This makes the code look like this:
246 GstMetaTiming *timing;
248 timing = gst_buffer_get_meta_timing (buffer);
249 timing->timestamp = 0;
250 timing->duration = 20 * GST_MSECOND;
252 To iterate the different metainfo structures, one can use the
253 gst_buffer_meta_get_next() methods.
255 GstMeta *current = NULL;
257 /* passing NULL gives the first entry */
258 current = gst_buffer_meta_get_next (buffer, current);
260 /* passing a GstMeta returns the next */
261 current = gst_buffer_meta_get_next (buffer, current);
269 We initially allocate a reasonable sized GstBuffer structure (say 512 bytes).
271 Since the complete buffer structure, including a large area for metadata, is
272 allocated in one go, we can reduce the number of memory allocations while still
273 providing dynamic metadata.
275 When adding metadata, we need to call the init function of the associated
276 metadata info structure. Since adding the metadata requires the caller to pass
277 a handle to the info, this operation does not require table lookups.
279 Per-metadata memory initialisation is needed because not all metadata is
280 initialized in the same way. We need to, for example, set the timestamps to
281 NONE in the MetaTiming structures.
283 The init/free functions can also be used to implement refcounting for a metadata
284 structure. This can be useful when a structure is shared between buffers.
286 When the free_size of the GstBuffer is exhausted, we will allocate new memory
287 for each newly added Meta and use the next pointers to point to this. It
288 is expected that this does not occur often and we might be able to optimize
289 this transparently in the future.
293 When a GstBuffer is freed, we potentially might have to call a custom free
294 function on the metadata info. In the case of the Memory metadata, we need to
295 call the associated free function to free the memory.
297 When freeing a GstBuffer, the custom buffer free function will iterate all of
298 the metadata in the buffer and call the associated free functions in the
299 MetaInfo associated with the entries. Usually, this function will be NULL.
305 When buffer should be sent over the wire or be serialized in GDP, we need a way
306 to perform custom serialization and deserialization on the metadata.
308 For this we add the serialize and deserialize functions to the metadata info.
309 Possible use cases are to make sure we write out the fields with a specific size
316 After certain transformations, the metadata on a buffer might not be relevant
319 Consider, for example, metadata that lists certain regions of interest
320 on the video data. If the video is scaled or rotated, the coordinates might not
321 make sense anymore. A transform element should be able to adjust or remove the
322 associated metadata when it becomes invalid.
324 We can make the transform element aware of the metadata so that it can adjust or
325 remove in an intelligent way. Since we allow arbitrary metadata, we can't do
326 this for all metadata and thus we need some other way.
328 One proposition is to tag the metadata type with keywords that specify what it
329 functionally refers too. We could, for example, tag the metadata for the regions
330 of interest with a tag that notes that the metadata refers to absolute pixel
331 positions. A transform could then know that the metadata is not valid anymore
332 when the position of the pixels changed (due to rotation, flipping, scaling and
339 Subbuffers are implemented with a generic copy. Parameters to the copy
340 are the offset and size. This allows each metadata structure to implement the
341 actions needed to update the metadata of the subbuffer.
343 It might not make sense for some metadata to work with subbuffers. For example
344 when we take a subbuffer of a buffer with a video frame, the GstMetaVideo
345 simply becomes invalid and is removed from the new subbuffer.
348 Relationship with GstCaps
349 ~~~~~~~~~~~~~~~~~~~~~~~~~
351 The difference between GstCaps, used in negotiation, and the metadata is not
354 We would like to think of the GstCaps containing the information needed to
355 functionally negotiate the format between two elements. The Metadata should then
356 only contain variables that can change between each buffer.
358 For example, for video we would have width/height/framerate in the caps but then
359 have the more technical details, such as stride, data pointers, pan/crop/zoom
362 A scheme like this would still allow us to functionally specify the desired
363 video resolution while the implementation details would be inside the metadata.
369 We need to make sure that elements exchange metadata that they both understand,
370 This is particulary important when the metadata describes the data layout in
371 memory (such as strides).
373 We would like to use the bufferpool negotiation system to negotiate the possible
374 metadata that can be exchanged between elements.
376 When deciding the allocation properties, we will also negotiate the buffer
377 metadata structures that we can exchange.
383 Some structures that we need to be able to add to buffers.
386 * Arbitrary Matrix Transform
391 Some of these overlap, we need to find a minimal set of metadata structures that
392 allows us to define all use cases.