4 This draft document describes a possible design for arbitrary per-buffer
7 The proposed changes in this document are not ABI/API compatible with the 0.10
8 version of GStreamer and should thus only be considered for upcomming unstable
11 Buffer metadata typically includes properties that give more information about
12 the buffer contents. These properties are usually not negotiated and are thus
15 Some examples of metadata:
19 - interlacing information
20 - video alignment, cropping, panning information
21 - extra container information such as granulepos, ...
22 - extra global buffer properties
29 * allocation, free, low fragmentation
30 * access to the metadata fields, preferably not much slower than directly
31 accessing a C structure field
32 - It must be extensible. Elements should be able to add new arbitrary metadata
33 without requiring much effort. Also new metadata fields should not break API
35 - It plays nice with subbuffers. When a subbuffer is created, the various
36 buffer metadata should be copied/updated correctly.
37 - We should be able to negotiate metadata between elements
44 Both DSP and CPU have separate MMUs and memory caches. When we exchange buffers
45 between two subsystems we need to flush caches so that one CPU can see the
46 modifications done by the other CPU. These cashes must only be flushed when one
47 CPU performed a write and the other CPU needs to do a read.
49 In order to implement this we need to be able to mark our read and write
50 operations on the buffer data.
57 We make GstMiniObject a simple refcounted C structure and also a GLib boxed
58 type. The following fields will be in the structure:
60 struct _GstMiniObject {
63 /*< public >*/ /* with COW */
69 GstMiniObjectCopyFunction copy;
70 GstMiniObjectDisposeFunction dispose;
71 GstMiniObjectFreeFunction free;
74 We will use the regular GSlice allocator or custom object pooling for allocating
75 instances of the mini object.
77 We use the well known refcounting mechanisms to manage the lifetime of the
81 GstEvent, GstCaps, GstQuery, GstMessage
82 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
84 Have the new GstMiniObject be the first field in these objects. They will probably
85 also replace the copy and free functions with their own implementations.
87 Allocation of the objects will use the regular gst_*_new() functions that will
88 allocate and initialize a parent GstMiniObject of the required size and setting up
95 A GstMiniObject will be the parent instance of the GstBuffer object, which is a
99 GstMiniObject mini_object;
104 GstBufferDataMap mmap_func;
105 GstBufferDataUnmap munmap_func;
107 gpointer _gst_padding[10];
110 The Buffer object will contain a pointer to the parent buffer to allow for subbuffers
111 as a first class feature of a GstBuffer.
113 Allocation of the GstBuffer structure will result in the allocation of a memory region
114 of a customizable size (512 bytes). Only the first sizeof (GstBuffer) bytes of this
115 region will initially be used. The remaining bytes will be part of the free metadata
116 region of the buffer. The size of the free region is kept in the free_size field.
118 Buffers point to a GstCaps structure that contains the caps of the buffer data.
120 We add two functions, map and unmap. We introduce explicit read and write transactions
121 on the buffer data in order to be able to implement flushing of the data between
122 different caches and mapping between different MMUs.
127 GST_BUFFER_MAP_WRITE,
130 gpointer gst_buffer_map (GstBuffer *, guint offset, guint *size, GstBufferMapFlags);
131 gboolean gst_buffer_unmap (GstBuffer *, gpointer data, guint size);
133 The map functions always create a contiguous memory space for the requested
134 area of the buffer. This might involve memcpy and other expensive operations. It
135 is assumed that more specialized elements can deal with the different memory
136 metadata structures in order to optimize access to the memory.
142 A GstMeta is a structure as follows:
145 GstMetaInfo *info; /* tag and info for the meta item */
148 The purpose of the this structure is to serve as a common header for all metadata
149 information that we can attach to a buffer. Specific metadata, such as timing metadata,
150 will have this structure as the first field. For example:
152 struct _GstMetaTiming {
153 GstMeta meta; /* common meta header */
155 GstClockTime dts; /* decoding timestamp */
156 GstClockTime pts; /* presentation timestamp */
157 GstClockTime duration; /* duration of the data */
158 GstClockTime clock_rate; /* clock rate for the above values */
161 Or another example for the buffer memory region that consists of some methods
164 struct _GstMetaMemory {
167 GstMetaMemoryMap mmap_func;
168 GstMetaMemoryUnmap munmap_func;
171 gpointer gst_meta_memory_map (GstMetaMemory *, guint offset, guint *size, GstBufferMapFlags);
172 gboolean gst_meta_memory_unmap (GstMetaMemory *, gpointer data, guint size);
175 GstMeta derived structures define the API of the metadata. The API can consist of
176 fields and/or methods. It is possible to have different implementations for the
177 same GstMeta structure.
179 The implementation of the GstMeta api would typically add more fields to the
180 public structure that allow it to implement the API. For example:
182 struct _GstMetaMemoryImpl {
183 GstMetaMemory memory;
193 GstMetaInfo will point to more information about the metadata and looks like this:
195 struct _GstMetaInfo {
196 GQuark api; /* api name */
197 GQuark impl; /* implementation name */
198 gsize size; /* size of the structure */
200 GstMetaInitFunction init_func;
201 GstMetaFreeFunction free_func;
202 GstMetaCopyFunction copy_func;
203 GstMetaSubFunction sub_func;
204 GstMetaSerializeFunction serialize_func
205 GstMetaDeserializeFunction deserialize_func
206 GstMetaConvFunction conv_func;
209 api will contain a GQuark of the metadata api. A repository of registered MetaInfo
210 will be maintained by the core. We will register some common metadata structures
211 in core and some media specific info for audio/video/text in -base. Plugins can
212 register additional custom metadata.
214 For each implementation of api, there will thus be a unique GstMetaInfo. In the
215 case of metadata with a well defined API, the implementation specific init
216 function will setup the methods in the metadata structure.
218 Along with the metadata description we will have functions to initialize/free (and/or refcount)
219 a specific GstMeta instance. We also have the possibility to add a custom subbuffer
220 function that can be used to modify the metadata when a subbuffer is taken.
222 We also add serialize and deserialize function for the metadata in case we need special
223 logic for reading and writing the metadata. This is needed for GDP payloading of the
226 We add a conv function to the Info structure that will be called when a buffer
227 should be converted to an old-style buffer for backward compatibility.
229 The purpose of the separate MetaInfo is to not have to carry the free/init functions in
230 each buffer instance but to define them globally. We still want quick access to the info
231 so we need to make the buffer metadata point to the info.
233 Technically we could also specify the field and types in the MetaInfo and
234 provide a generic API to retrieve the metadata fields without the need for a
235 header file. We will not do this yet.
237 The complete buffer with metadata would then look as follows:
239 +-------------------------------------+
240 GstMiniObject | GType (GstBuffer) |
241 | refcount, flags, copy/disp/free |
242 +-------------------------------------+
243 GstBuffer | caps, parent |
244 +.....................................+
246 +- | info ------> GstMetaInfo
247 GstMetaTiming | | | |
252 + . . . . . . . . . . . . . . . . . . + |
254 GstMetaMemory +- +- | info ------> GstMetaInfo
260 GstMetaMemoryImpl | | size | |
264 + . . . . . . . . . . . . . . . . . . + .
271 Buffers are created using the normal gst_buffer_new functions. The standard fields
272 are initialized as usual. A memory area that is bigger than the structure size
273 is allocated for the buffer metadata.
277 After creating a buffer, the application can set caps and add metadata
280 To add or retrieve metadata, a handle to a GstMetaInfo structure needs to be
281 obtained. This defines the implementation and API of the metadata. Usually, a
282 handle to this info structure can be obtained by callnig a public _get_info()
283 method from a shared library (for shared metadata).
285 The following defines can usually be found in the shared .h file.
287 GstMetaInfo * gst_meta_timing_get_info();
288 #define GST_META_TIMING_INFO (gst_meta_timing_get_info())
290 Retrieving and/or creating the metadata on a buffer can be done with the
291 gst_buffer_meta_get() method. This function retrieves an existing metadata
292 conforming to the API spcified in the given info. When no such metadata exists
293 and the last gboolean argument is true, a new metadata item will be created from
294 the info and added to the buffer.
296 GstMetaTiming *timing;
298 timing = gst_buffer_get_meta (buffer, GST_META_TIMING_INFO, TRUE);
300 Once a reference to the info has been obtained, the associated metadata can be
301 added or modified on a buffer.
303 timing->timestamp = 0;
304 timing->duration = 20 * GST_MSECOND;
306 Other convenience macros can be made to simplify the above code:
308 #define gst_buffer_get_meta_timing(b,c) \
309 ((GstMetaTiming *) gst_buffer_get_meta ((b), GST_META_TIMING_INFO, (c))
311 This makes the code look like this:
313 GstMetaTiming *timing;
315 timing = gst_buffer_get_meta_timing (buffer, TRUE); /* TRUE = create if absent */
316 timing->timestamp = 0;
317 timing->duration = 20 * GST_MSECOND;
320 We will also provide an API to iterate the different metainfo structures. A
321 possible simple API would look like this:
323 GstMeta *current = NULL;
325 /* passing NULL gives the first entry */
326 current = gst_buffer_meta_get_next (buffer, current);
328 /* passing a GstMeta returns the next */
329 current = gst_buffer_meta_get_next (buffer, current);
338 We will initially allocate a reasonable sized GstBuffer structure (say 512
341 Since the complete buffer structure, including a large area for metadata, is
342 allocated in one go, we can reduce the number of memory allocations while still
343 providing dynamic metadata.
345 When adding metadata, we need to call the init function of the associated
346 metadata info structure. Since adding the metadata requires the caller to pass
347 a handle to the info, this operation does not require table lookups.
349 Per-metadata memory initialisation is needed because not all metadata is
350 initialized in the same way. We need to, for example, set the timestamps to
351 NONE in the MetaTiming structures.
353 The init/free functions can also be used to implement refcounting for a metadata
354 structure. This can be useful when a structure is shared between buffers.
356 When the free_size of the GstBuffer is exhausted, we will allocate new memory
357 for each newly added Meta and use the next pointers to point to this. It
358 is expected that this does not occur often and we might be able to optimize
359 this transparently in the future.
363 When a GstBuffer is freed, we potentially might have to call a custom free
364 function on the metadata info. In the case of the Memory metadata, we need to
365 call the associated free function to free the memory.
367 When freeing a GstBuffer, the custom buffer free function will iterate all of
368 the metadata in the buffer and call the associated free functions in the
369 MetaInfo associated with the entries. Usually, this function will be NULL.
375 Subbuffers are a first class feature of the GstBuffer.
377 Creating a subbuffer from a GstBuffer will allocate a new GstBuffer and ref the
378 parent buffer. It will then iterate all of the metadata entries for the parent
379 buffer and call the associated sub_func in the MetaInfo.
381 This allows each metadata structure to implement the actions needed to update
382 the metadata of the subbuffer.
384 A pointer to the old and new memory location of the metadata is passed to the
385 sub_func. The default implementation will simply copy the metadata. Custom
386 implementations can adjust the values. For example, when making a subbuffer, the
387 timing metadata needs to be reset to NONE when the start offset is different.
393 When buffer should be sent over the wire or be serialized in GDP, we need a way
394 to perform custom serialization and deserialization on the metadata.
396 For this we add the serialize and deserialize functions to the metadata info.
397 Possible use cases are to make sure we write out the fields with a specific size
404 After certain transformations, the metadata on a buffer might not be relevant
407 Consider, for example, metadata that lists certain regions of interest
408 on the video data. If the video is scaled or rotated, the coordinates might not
409 make sense anymore. A transform element should be able to adjust or remove the
410 associated metadata when it becomes invalid.
412 We can make the transform element aware of the metadata so that it can adjust or
413 remove in an intelligent way. Since we allow arbitrary metadata, we can't do
414 this for all metadata and thus we need some other way.
416 One proposition is to tag the metadata type with keywords that specify what it
417 functionally refers too. We could, for example, tag the metadata for the regions
418 of interest with a tag that notes that the metadata refers to absolute pixel
419 positions. A transform could then know that the metadata is not valid anymore
420 when the position of the pixels changed (due to rotation, flipping, scaling and
427 Making the GstMetaMemory (for making the buffer point to the associated
428 memory region) as metadata on a GstBuffer, as opposed to making it an integral
429 part of GstBuffer, allows for some more interesting ways to transfer data.
431 We could for example make a new GstMetaIOVec metadata structure like this:
433 struct _GstMetaIOVec {
436 /* pointer to data and its size */
443 This would allow us to transfer data in a scatter/gather array. Since the fields
444 in the buffer metadata are now explicit, elements that don't support this kind
445 of metadata can gracefully degrade.
447 Another use case for not having the Memory metadata in the buffers would be for
448 _pad_alloc() and get_range(). We can pass a GstBuffer with the requested
449 metadata fields to those functions and have the _get_range() or pad_alloc()
450 implementations add (or use, in the case of a file reader) the memory metadata.
453 Relationship with GstCaps
454 ~~~~~~~~~~~~~~~~~~~~~~~~~
456 The difference between GstCaps, used in negotiation, and the metadata is not
459 We would like to think of the GstCaps containing the information needed to
460 functionally negotiate the format between two elements. The Metadata should then
461 only contain variables that can change between each buffer.
463 For example, for video we would have width/height/framerate in the caps but then
464 have the more technical details, such as stride, data pointers, pan/crop/zoom
467 A scheme like this would still allow us to functionally specify the desired
468 video resolution while the implementation details would be inside the metadata.
474 We need to make sure that elements exchange metadata that they both understand,
475 This is particulary important when the metadata describes the data layout in
476 memory (such as strides).
478 We would like to use the bufferpool negotiation system to negotiate the possible
479 metadata that can be exchanged between elements.
481 When deciding the allocation properties, we will also negotiate the buffer
482 metadata structures that we can exchange.
488 Some structures that we need to be able to add to buffers.
491 * Arbitrary Matrix Transform
496 Some of these overlap, we need to find a minimal set of metadata structures that
497 allows us to define all use cases.
504 #define GST_VIDEO_MAX_PLANES 4
506 struct GstVideoPlane {
517 struct GstBufferVideoMeta {
520 GstBufferVideoFlags flags
523 GstVideoPlane plane[GST_VIDEO_MAX_PLANES];