4 This draft document describes a possible design for arbitrary per-buffer
7 The proposed changes in this document are not ABI/API compatible with the 0.10
8 version of GStreamer and should thus only be considered for upcoming unstable
11 Buffer metadata typically includes properties that give more information about
12 the buffer contents. These properties are usually not negotiated and are thus
15 Some examples of metadata:
19 - interlacing information
20 - video alignment, cropping, panning information
21 - extra container information such as granulepos, ...
22 - extra global buffer properties
29 * allocation, free, low fragmentation
30 * access to the metadata fields, preferably not much slower than directly
31 accessing a C structure field
32 - It must be extensible. Elements should be able to add new arbitrary metadata
33 without requiring much effort. Also new metadata fields should not break API
35 - It plays nice with subbuffers. When a subbuffer is created, the various
36 buffer metadata should be copied/updated correctly.
37 - We should be able to pass metadata in pad_alloc() and get_range() functions
38 to specify extra allocation parameters.
39 - We should be able to attach statically allocated metadata to a buffer. This
40 is for metadata that does not change much.
46 We make GstMiniObject a simple refcounted C structure and also a GLib boxed
47 type. The following fields will be in the structure:
49 struct _GstMiniObject {
52 /*< public >*/ /* with COW */
57 GstMiniObjectCopyFunction copy;
58 GstMiniObjectFreeFunction free;
61 We will use the regular GSlice allocator or custom object pooling for allocating
62 instances of the mini object.
64 We use the well known refcounting mechanisms to manage the lifetime of the
68 GstEvent, GstCaps, GstQuery, GstMessage
69 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
71 Have the new GstMiniObject be the first field in these objects. They will probably
72 also replace the copy and free functions with their own implementations.
74 Allocation of the objects will use the regular gst_*_new() functions that will
75 allocate and initialize a parent GstMiniObject of the required size and setting up
82 A GstMiniObject will be the parent instance of the GstBuffer object, which is a
86 GstMiniObject mini_object;
93 gpointer _gst_padding[10];
96 The Buffer object will contain a pointer to the parent buffer to allow for subbuffers
97 as a first class feature of a GstBuffer.
99 Allocation of the GstBuffer structure will result in the allocation of a memory region
100 of a customizable size (512 bytes). Only the first sizeof (GstBuffer) bytes of this
101 region will initially be used. The remaining bytes will be part of the free metadata
102 region of the buffer. The size of the free region is kept in the free_size field.
104 Buffers point to a GstCaps structure that contains the caps of the buffer data.
110 A GstBufferMeta is a structure as follows:
112 struct _GstBufferMeta {
113 GstBufferMetaInfo *info; /* tag and info for the meta item */
114 GstBufferMeta *next; /* pointer to the next item */
117 The purpose of the this structure is to serve as a common header for all metadata
118 information that we can attach to a buffer. Specific metadata, such as timing metadata,
119 will have this structure as the first field. For example:
121 struct _GstBufferMetaTiming {
122 GstBufferMeta meta; /* common meta header */
124 GstClockTime dts; /* decoding timestamp */
125 GstClockTime pts; /* presentation timestamp */
126 GstClockTime duration; /* duration of the data */
127 GstClockTime clock_rate; /* clock rate for the above values */
130 Or another example for the buffer memory region
132 struct _GstBufferMetaMemory {
135 /* pointer to data and its size */
143 GstBufferMetaInfo will point to more information about the metadata and looks like this:
145 struct _GstBufferMetaInfo {
146 GQuark tag; /* tag name */
147 gsize size; /* size of the structure */
149 GstMetaInitFunction init_func;
150 GstMetaFreeFunction free_func;
151 GstMetaCopyFunction copy_func;
152 GstMetaSubFunction sub_func;
153 GstMetaSerializeFunction serialize_func
154 GstMetaDeserializeFunction deserialize_func
155 GstMetaConvFunction conv_func;
158 Tag will contain a GQuark of the metadata name. We will be able to refer to specific
159 metadata by name or by its (cached) GQuark. A repository of registered MetaInfo
160 will be maintained by the core. We will register some common metadata structures
161 in core and some media specific info for audio/video/text in -base. Plugins can
162 register additional custom metadata.
164 Along with the metadata description we will have functions to initialize/free (and/or refcount)
165 a specific GstBufferMeta instance. We also have the possibility to add a custom subbuffer
166 function that can be used to modify the metadata when a subbuffer is taken.
168 We also add serialize and deserialize function for the metadata in case we need special
169 logic for reading and writing the metadata. This is needed for GDP payloading of the
172 We add a conv function to the Info structure that will be called when a buffer
173 should be converted to an old-style buffer for backward compatibility.
175 The purpose of the separate MetaInfo is to not have to carry the free/init functions in
176 each buffer instance but to define them globally. We still want quick access to the info
177 so we need to make the buffer metadata point to the info.
179 Technically we could also specify the field and types in the MetaInfo and
180 provide a generic API to retrieve the metadata fields without the need for a
181 header file. We will not do this yet.
183 The complete buffer with metadata would then look as follows:
185 +-------------------------------------+
186 GstMiniObject | GType (GstBuffer) |
187 | refcount, flags, copy/free |
188 +-------------------------------------+
189 GstBuffer | caps, parent, subfunc |
190 +.....................................+
191 +- | info ------> GstBufferMetaInfo
192 GstBufferMetaTiming | | next ---+
198 + . . . . . . . . . . . . . . . . . . + |
199 +- | info <--+ -> GstBufferMetaInfo
200 GstBufferMetaMemory | | next ---+
207 + . . . . . . . . . . . . . . . . . . + .
213 Buffers are created using the normal gst_buffer_new functions. The standard fields
214 are initialized as usual. A memory area that is bigger than the structure size
215 is allocated for the buffer metadata. The remaining free area is stored in the
220 After creating a buffer, the application can set caps. and add other metadata
223 In order to modify metadata, a reference to the MetaInfo should be obtained.
224 This can be done like this:
226 GstBufferMetaInfo *info;
228 info = gst_buffer_meta_get_info (GQuark tag);
230 Usually the info will be obtained only once in order to avoid lock contention on
231 the global pool of meta info. The core will also provide convenience functions
232 for the core metainfo.
234 Once a reference to the info has been obtained, the associated metadata can be
235 added or modified on a buffer.
237 For example, to modify the timing info on a buffer, one could use the following
240 GstBufferMetaInfo *info;
241 GstBufferMetaTiming *timing;
243 info = gst_buffer_meta_get_info (GST_META_TIMING_QUARK);
245 timing = gst_buffer_get_meta (buffer, info, TRUE); /* TRUE = create if absent */
246 timing->timestamp = 0;
247 timing->duration = 20 * GST_MSECOND;
249 The _get_meta() function returns a pointer to the metadata structure associated
250 with the GST_META_TIMING_QUARK info.
252 For the core meta info, we will provide convenience code that uses the cached
253 GstBufferMetaInfo, making the above code a little more simple.
255 GstBufferMetaTiming *timing;
257 timing = gst_buffer_get_meta_timing (buffer, TRUE); /* TRUE = create if absent */
258 timing->timestamp = 0;
259 timing->duration = 20 * GST_MSECOND;
261 Note that for each of the metadata that we will add to buffers, we need a struct
262 definition and a registered MetaInfo.
265 We will also provide an API to iterate the different metainfo structures. A
266 possible simple API would look like this:
268 GstBufferMeta *current = NULL;
270 /* passing NULL gives the first entry */
271 current = gst_buffer_meta_get_next (buffer, current);
273 /* passing a GstBufferMeta returns the next */
274 current = gst_buffer_meta_get_next (buffer, current);
283 We will initially allocate a reasonable sized GstBuffer structure (say 512
284 bytes) and we will set the free_size to the maximum amount of metadata we can
287 Since the complete buffer structure, including a large area for metadata, is
288 allocated in one go, we can reduce the number of memory allocations while still
289 providing dynamic metadata.
291 When adding metadata, we need to call the init function of the associated
292 metadata info structure. Since adding the metadata requires the caller to pass
293 a handle to the info, this operation does not require table lookups.
295 Per-metadata memory initialisation is needed because not all metadata is
296 initialized in the same way. We need to, for example, set the timestamps to
297 NONE in the MetaTiming structures.
299 The init/free functions can also be used to implement refcounting for a metadata
300 structure. This can be useful when a structure is shared between buffers.
302 When the free_size of the GstBuffer is exhausted, we will allocate new memory
303 for each newly added BufferMeta and use the next pointers to point to this. It
304 is expected that this does not occur often and we might be able to optimize
305 this transparently in the future.
309 When a GstBuffer is freed, we potentially might have to call a custom free
310 function on the metadata info. In the case of the Memory metadata, we need to
311 call the associated free function to free the memory.
313 When freeing a GstBuffer, the custom buffer free function will iterate all of
314 the metadata in the buffer and call the associated free functions in the
315 MetaInfo associated with the entries. Usually, this function will be NULL.
321 Subbuffers are a first class feature of the GstBuffer.
323 Creating a subbuffer from a GstBuffer will allocate a new GstBuffer and ref the
324 parent buffer. It will then iterate all of the metadata entries for the parent
325 buffer and call the associated sub_func in the MetaInfo.
327 This allows each metadata structure to implement the actions needed to update
328 the metadata of the subbuffer.
330 A pointer to the old and new memory location of the metadata is passed to the
331 sub_func. The default implementation will simply copy the metadata. Custom
332 implementations can adjust the values. For example, when making a subbuffer, the
333 timing metadata needs to be reset to NONE when the start offset is different.
339 When buffer should be sent over the wire or be serialized in GDP, we need a way
340 to perform custom serialization and deserialization on the metadata.
342 For this we add the serialize and deserialize functions to the metadata info.
343 Possible use cases are to make sure we write out the fields with a specific size
350 After certain transformations, the metadata on a buffer might not be relevant
353 Consider, for example, metadata that lists certain regions of interest
354 on the video data. If the video is scaled or rotated, the coordinates might not
355 make sense anymore. A transform element should be able to adjust or remove the
356 associated metadata when it becomes invalid.
358 We can make the transform element aware of the metadata so that it can adjust or
359 remove in an intelligent way. Since we allow arbitrary metadata, we can't do
360 this for all metadata and thus we need some other way.
362 One proposition is to tag the metadata type with keywords that specify what it
363 functionally refers too. We could, for example, tag the metadata for the regions
364 of interest with a tag that notes that the metadata refers to absolute pixel
365 positions. A transform could then know that the metadata is not valid anymore
366 when the position of the pixels changed (due to rotation, flipping, scaling and
373 Making the GstBufferMetaMemory (for making the buffer point to the associated
374 memory region) as metadata on a GstBuffer, as opposed to making it an integral
375 part of GstBuffer, allows for some more interesting ways to transfer data.
377 We could for example make a new GstBufferMetaIOVec metadata structure like this:
379 struct _GstBufferMetaIOVec {
382 /* pointer to data and its size */
389 This would allow us to transfer data in a scatter/gather array. Since the fields
390 in the buffer metadata are now explicit, elements that don't support this kind
391 of metadata can gracefully degrade.
393 Another use case for not having the Memory metadata in the buffers would be for
394 _pad_alloc() and get_range(). We can pass a GstBuffer with the requested
395 metadata fields to those functions and have the _get_range() or pad_alloc()
396 implementations add (or use, in the case of a file reader) the memory metadata.
399 Relationship with GstCaps
400 ~~~~~~~~~~~~~~~~~~~~~~~~~
402 The difference between GstCaps, used in negotiation, and the metadata is not
405 We would like to think of the GstCaps containing the information needed to
406 functionally negotiate the format between two elements. The Metadata should then
407 only contain variables that can change between each buffer.
409 For example, for video we would have width/height/framerate in the caps but then
410 have the more technical details, such as stride, data pointers, pan/crop/zoom
413 A scheme like this would still allow us to functionally specify the desired
414 video resolution while the implementation details would be inside the metadata.
420 We need to make sure that elements exchange metadata that they both understand,
421 This is particulary important when the metadata describes the data layout in
422 memory (such as strides).
424 Currently the only way to communicate buffer formats between elements is by
425 using caps. We would like to use the caps system to negotiate the metadata that
426 will be put on buffers.
428 We would like to add to the caps on the buffer (and pad) an array of metadata
429 structures (as strings) that is on the buffer. This way, an element can
430 quickly know what metadata to look for.
432 In order to remain compatibility with older plugins we need to convert buffers
433 that use metadata to specify a non-standard data layout to the old format. We
434 need to do this before handing buffers to old elements. We will require elements
435 that are metadata aware to set a flag on their pads; any buffer passed on that
436 pad will be converted to the old layout when the flag is not set.
442 Some structures that we need to be able to add to buffers.
445 * Arbitrary Matrix Transform
450 Some of these overlap, we need to find a minimal set of metadata structures that
451 allows us to define all use cases.