+++ /dev/null
-GstBuffer^2
------------
-
-This draft document describes a possible design for arbitrary per-buffer
-metadata.
-
-The proposed changes in this document are not ABI/API compatible with the 0.10
-version of GStreamer and should thus only be considered for upcomming unstable
-versions.
-
-Buffer metadata typically includes properties that give more information about
-the buffer contents. These properties are usually not negotiated and are thus
-not inside the caps.
-
-Some examples of metadata:
-
- - timestamp, duration
- - offset, offset_end
- - interlacing information
- - video alignment, cropping, panning information
- - extra container information such as granulepos, ...
- - extra global buffer properties
-
-
-Requirements
-~~~~~~~~~~~~
-
- - It must be fast
- * allocation, free, low fragmentation
- * access to the metadata fields, preferably not much slower than directly
- accessing a C structure field
- - It must be extensible. Elements should be able to add new arbitrary metadata
- without requiring much effort. Also new metadata fields should not break API
- or ABI.
- - It plays nice with subbuffers. When a subbuffer is created, the various
- buffer metadata should be copied/updated correctly.
- - We should be able to negotiate metadata between elements
-
-Use cases
----------
-
- * DSP vs CPU caches
-
- Both DSP and CPU can have separate MMUs and memory caches. When we exchange buffers
- between two subsystems we need to flush caches so that one CPU can see the
- modifications done by the other CPU. These cashes must only be flushed when one
- CPU performed a write and the other CPU needs to do a read.
-
- In order to implement this we need to be able to mark our read and write
- operations on the buffer data.
-
- It might also be possible that buffers are not mapped into the address space of
- the process normally and that an explicit mmap operation is needed to setup
- the mapping tables for the physical memory.
-
- * Video planes
-
- Video data is sometimes allocated in non-contiguous planes for the Y and the UV
- data. We need to be able to specify the data on a buffer using multiple
- pointers in memory. We also need to be able to specify the stride for these
- planes.
-
- * Extra buffer data
-
- Some elements might need to store extra data for a buffer. This is typically
- done when the resources are allocated from another subsystem such as OMX or
- X11.
-
- * Processing information
-
- Pan and crop information can be added to the buffer data when the downstream
- element can understand and use this metadata. An imagesink can, for example,
- use the pan and cropping formation when it composites the image on the screen
- with little overhead.
-
-
-GstMiniObject
-~~~~~~~~~~~~~
-
-We make GstMiniObject a simple refcounted C structure and also a GLib boxed
-type. The following fields will be in the structure:
-
-struct _GstMiniObject {
- GType type;
-
- /*< public >*/ /* with COW */
- /* refcounting */
- gint refcount;
- guint flags;
- gsize size;
-
- GstMiniObjectCopyFunction copy;
- GstMiniObjectDisposeFunction dispose;
- GstMiniObjectFreeFunction free;
-}
-
-We will use the regular GSlice allocator or custom object pooling for allocating
-instances of the mini object.
-
-We use the well known refcounting mechanisms to manage the lifetime of the
-objects.
-
-
-GstEvent, GstCaps, GstQuery, GstMessage
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Have the new GstMiniObject be the first field in these objects. They will probably
-also replace the copy and free functions with their own implementations.
-
-Allocation of the objects will use the regular gst_*_new() functions that will
-allocate and initialize a parent GstMiniObject of the required size and setting up
-the custom functions.
-
-
-GstBuffer
-~~~~~~~~~
-
-A GstMiniObject will be the parent instance of the GstBuffer object, which is a
-regular C structure.
-
-struct _GstBuffer {
- GstMiniObject mini_object;
-
- GstCaps *caps;
- GstBuffer *parent;
- GstBufferPool *pool;
-};
-
-The Buffer object will contain a pointer to the parent buffer to allow for subbuffers
-as a first class feature of a GstBuffer.
-
-Allocation of the GstBuffer structure will result in the allocation of a memory region
-of a customizable size (512 bytes). Only the first sizeof (GstBuffer) bytes of this
-region will initially be used. The remaining bytes will be part of the free metadata
-region of the buffer. The size of the free region is kept in the free_size field.
-
-Buffers point to a GstCaps structure that contains the caps of the buffer data.
-
-The buffer contains GstMeta items describing the memory is wraps along with
-metadata for the timing information etc.
-
-
-GstMeta
-~~~~~~~
-
-A GstMeta is a structure as follows:
-
- struct _GstMeta {
- GstMetaInfo *info; /* tag and info for the meta item */
- };
-
-The purpose of the this structure is to serve as a common header for all metadata
-information that we can attach to a buffer. Specific metadata, such as timing metadata,
-will have this structure as the first field. For example:
-
- struct _GstMetaTiming {
- GstMeta meta; /* common meta header */
-
- GstClockTime dts; /* decoding timestamp */
- GstClockTime pts; /* presentation timestamp */
- GstClockTime duration; /* duration of the data */
- GstClockTime clock_rate; /* clock rate for the above values */
- };
-
-Or another example for the buffer memory region that consists of some methods
-only.
-
- struct _GstMetaMemory {
- GstMeta meta;
-
- GstMetaMemoryMap mmap_func;
- GstMetaMemoryUnmap munmap_func;
- };
-
- typedef enum {
- GST_BUFFER_MAP_NONE,
- GST_BUFFER_MAP_READ,
- GST_BUFFER_MAP_WRITE,
- } GstBufferMapFlags
-
- gpointer gst_meta_memory_map (GstMetaMemory *, guint offset, guint *size, GstBufferMapFlags);
- gboolean gst_meta_memory_unmap (GstMetaMemory *, gpointer data, guint size);
-
-
-GstMeta derived structures define the API of the metadata. The API can consist of
-fields and/or methods. It is possible to have different implementations for the
-same GstMeta structure.
-
-The implementation of the GstMeta api would typically add more fields to the
-public structure that allow it to implement the API. For example:
-
- struct _GstMetaMemoryImpl {
- GstMetaMemory memory;
-
- gpointer *data;
- guint size;
- gpointer *data_orig;
- GFreeFunc data_free;
- gpointer data_user;
- };
-
-
-GstMetaInfo will point to more information about the metadata and looks like this:
-
- struct _GstMetaInfo {
- GQuark api; /* api name */
- GQuark impl; /* implementation name */
- gsize size; /* size of the structure */
-
- GstMetaInitFunction init_func;
- GstMetaFreeFunction free_func;
- GstMetaCopyFunction copy_func;
- GstMetaSubFunction sub_func;
- GstMetaSerializeFunction serialize_func
- GstMetaDeserializeFunction deserialize_func
- GstMetaConvFunction conv_func;
- };
-
-api will contain a GQuark of the metadata api. A repository of registered MetaInfo
-will be maintained by the core. We will register some common metadata structures
-in core and some media specific info for audio/video/text in -base. Plugins can
-register additional custom metadata.
-
-For each implementation of api, there will thus be a unique GstMetaInfo. In the
-case of metadata with a well defined API, the implementation specific init
-function will setup the methods in the metadata structure.
-
-Along with the metadata description we will have functions to initialize/free (and/or refcount)
-a specific GstMeta instance. We also have the possibility to add a custom subbuffer
-function that can be used to modify the metadata when a subbuffer is taken.
-
-We also add serialize and deserialize function for the metadata in case we need special
-logic for reading and writing the metadata. This is needed for GDP payloading of the
-metadata.
-
-We add a conv function to the Info structure that will be called when a buffer
-should be converted to an old-style buffer for backward compatibility.
-
-The purpose of the separate MetaInfo is to not have to carry the free/init functions in
-each buffer instance but to define them globally. We still want quick access to the info
-so we need to make the buffer metadata point to the info.
-
-Technically we could also specify the field and types in the MetaInfo and
-provide a generic API to retrieve the metadata fields without the need for a
-header file. We will not do this yet.
-
-The complete buffer with metadata would then look as follows:
-
- +-------------------------------------+
-GstMiniObject | GType (GstBuffer) |
- | refcount, flags, copy/disp/free |
- +-------------------------------------+
-GstBuffer | caps, parent, pool |
- +.....................................+
- | next ---+
- +- | info ------> GstMetaInfo
-GstMetaTiming | | | |
- | | dts | |
- | | pts | |
- | | duration | |
- +- | clock_rate | |
- + . . . . . . . . . . . . . . . . . . + |
- | next <--+
-GstMetaMemory +- +- | info ------> GstMetaInfo
- | | | | |
- | | | mmap | |
- | | | munmap | |
- +- | | | |
- | | data | |
-GstMetaMemoryImpl | | size | |
- | | mallocdata | |
- | | data_free | |
- +- | data_user | |
- + . . . . . . . . . . . . . . . . . . + .
- . .
-
-
-API examples
-~~~~~~~~~~~~
-
-Buffers are created using the normal gst_buffer_new functions. The standard fields
-are initialized as usual. A memory area that is bigger than the structure size
-is allocated for the buffer metadata.
-
- gst_buffer_new ();
-
-After creating a buffer, the application can set caps and add metadata
-information.
-
-To add or retrieve metadata, a handle to a GstMetaInfo structure needs to be
-obtained. This defines the implementation and API of the metadata. Usually, a
-handle to this info structure can be obtained by callnig a public _get_info()
-method from a shared library (for shared metadata).
-
-The following defines can usually be found in the shared .h file.
-
- GstMetaInfo * gst_meta_timing_get_info();
- #define GST_META_TIMING_INFO (gst_meta_timing_get_info())
-
-Retrieving and/or creating the metadata on a buffer can be done with the
-gst_buffer_meta_get() method. This function retrieves an existing metadata
-conforming to the API spcified in the given info. When no such metadata exists
-and the last gboolean argument is true, a new metadata item will be created from
-the info and added to the buffer.
-
- GstMetaTiming *timing;
-
- timing = gst_buffer_get_meta (buffer, GST_META_TIMING_INFO, TRUE);
-
-Once a reference to the info has been obtained, the associated metadata can be
-added or modified on a buffer.
-
- timing->timestamp = 0;
- timing->duration = 20 * GST_MSECOND;
-
-Other convenience macros can be made to simplify the above code:
-
- #define gst_buffer_get_meta_timing(b,c) \
- ((GstMetaTiming *) gst_buffer_get_meta ((b), GST_META_TIMING_INFO, (c))
-
-This makes the code look like this:
-
- GstMetaTiming *timing;
-
- timing = gst_buffer_get_meta_timing (buffer, TRUE); /* TRUE = create if absent */
- timing->timestamp = 0;
- timing->duration = 20 * GST_MSECOND;
-
-
-We will also provide an API to iterate the different metainfo structures. A
-possible simple API would look like this:
-
- GstMeta *current = NULL;
-
- /* passing NULL gives the first entry */
- current = gst_buffer_meta_get_next (buffer, current);
-
- /* passing a GstMeta returns the next */
- current = gst_buffer_meta_get_next (buffer, current);
-
-
-
-Memory management
-~~~~~~~~~~~~~~~~~
-
-* allocation
-
- We will initially allocate a reasonable sized GstBuffer structure (say 512
- bytes).
-
- Since the complete buffer structure, including a large area for metadata, is
- allocated in one go, we can reduce the number of memory allocations while still
- providing dynamic metadata.
-
- When adding metadata, we need to call the init function of the associated
- metadata info structure. Since adding the metadata requires the caller to pass
- a handle to the info, this operation does not require table lookups.
-
- Per-metadata memory initialisation is needed because not all metadata is
- initialized in the same way. We need to, for example, set the timestamps to
- NONE in the MetaTiming structures.
-
- The init/free functions can also be used to implement refcounting for a metadata
- structure. This can be useful when a structure is shared between buffers.
-
- When the free_size of the GstBuffer is exhausted, we will allocate new memory
- for each newly added Meta and use the next pointers to point to this. It
- is expected that this does not occur often and we might be able to optimize
- this transparently in the future.
-
-* free
-
- When a GstBuffer is freed, we potentially might have to call a custom free
- function on the metadata info. In the case of the Memory metadata, we need to
- call the associated free function to free the memory.
-
- When freeing a GstBuffer, the custom buffer free function will iterate all of
- the metadata in the buffer and call the associated free functions in the
- MetaInfo associated with the entries. Usually, this function will be NULL.
-
-
-Subbuffers
-~~~~~~~~~~
-
-Subbuffers are a first class feature of the GstBuffer.
-
-Creating a subbuffer from a GstBuffer will allocate a new GstBuffer and ref the
-parent buffer. It will then iterate all of the metadata entries for the parent
-buffer and call the associated sub_func in the MetaInfo.
-
-This allows each metadata structure to implement the actions needed to update
-the metadata of the subbuffer.
-
-A pointer to the old and new memory location of the metadata is passed to the
-sub_func. The default implementation will simply copy the metadata. Custom
-implementations can adjust the values. For example, when making a subbuffer, the
-timing metadata needs to be reset to NONE when the start offset is different.
-
-Since the subbuffer expects an offset and size, it might not make sense to make
-subbuffers from arbitrary buffers. Video metadata that has data in muliple
-planes, for example, might need to copy the planes to its 'natural' contiguous
-representation for the subbuffer.
-
-
-Serialization
-~~~~~~~~~~~~~
-
-When buffer should be sent over the wire or be serialized in GDP, we need a way
-to perform custom serialization and deserialization on the metadata.
-
-For this we add the serialize and deserialize functions to the metadata info.
-Possible use cases are to make sure we write out the fields with a specific size
-and endianness.
-
-
-Transformations
-~~~~~~~~~~~~~~~
-
-After certain transformations, the metadata on a buffer might not be relevant
-anymore.
-
-Consider, for example, metadata that lists certain regions of interest
-on the video data. If the video is scaled or rotated, the coordinates might not
-make sense anymore. A transform element should be able to adjust or remove the
-associated metadata when it becomes invalid.
-
-We can make the transform element aware of the metadata so that it can adjust or
-remove in an intelligent way. Since we allow arbitrary metadata, we can't do
-this for all metadata and thus we need some other way.
-
-One proposition is to tag the metadata type with keywords that specify what it
-functionally refers too. We could, for example, tag the metadata for the regions
-of interest with a tag that notes that the metadata refers to absolute pixel
-positions. A transform could then know that the metadata is not valid anymore
-when the position of the pixels changed (due to rotation, flipping, scaling and
-so on).
-
-
-Other use cases
-~~~~~~~~~~~~~~~
-
-Making the GstMetaMemory (for making the buffer point to the associated
-memory region) as metadata on a GstBuffer, as opposed to making it an integral
-part of GstBuffer, allows for some more interesting ways to transfer data.
-
-We could for example make a new GstMetaIOVec metadata structure like this:
-
- struct _GstMetaIOVec {
- GstMeta meta;
-
- /* pointer to data and its size */
- GFreeFunc data_free;
- gpointer data_user;
- guint len;
- struct iovec *iov;
- };
-
-This would allow us to transfer data in a scatter/gather array. Since the fields
-in the buffer metadata are now explicit, elements that don't support this kind
-of metadata can gracefully degrade.
-
-Another use case for not having the Memory metadata in the buffers would be for
-_pad_alloc() and get_range(). We can pass a GstBuffer with the requested
-metadata fields to those functions and have the _get_range() or pad_alloc()
-implementations add (or use, in the case of a file reader) the memory metadata.
-
-
-Relationship with GstCaps
-~~~~~~~~~~~~~~~~~~~~~~~~~
-
-The difference between GstCaps, used in negotiation, and the metadata is not
-clearly defined.
-
-We would like to think of the GstCaps containing the information needed to
-functionally negotiate the format between two elements. The Metadata should then
-only contain variables that can change between each buffer.
-
-For example, for video we would have width/height/framerate in the caps but then
-have the more technical details, such as stride, data pointers, pan/crop/zoom
-etc in the metadata.
-
-A scheme like this would still allow us to functionally specify the desired
-video resolution while the implementation details would be inside the metadata.
-
-
-Compatibility
-~~~~~~~~~~~~~
-
-We need to make sure that elements exchange metadata that they both understand,
-This is particulary important when the metadata describes the data layout in
-memory (such as strides).
-
-We would like to use the bufferpool negotiation system to negotiate the possible
-metadata that can be exchanged between elements.
-
-When deciding the allocation properties, we will also negotiate the buffer
-metadata structures that we can exchange.
-
-
-Notes
-~~~~~
-
-Some structures that we need to be able to add to buffers.
-
-* Clean Aperture
-* Arbitrary Matrix Transform
-* Aspect ratio
-* Pan/crop/zoom
-* Video strides
-
-Some of these overlap, we need to find a minimal set of metadata structures that
-allows us to define all use cases.
-
-
-
-Video Buffers
--------------
-
- #define GST_VIDEO_MAX_PLANES 4
-
- struct GstVideoPlane {
- guint8 *data;
- guint size;
- guint stride;
-
- guint8 *data_orig;
- guint size_orig;
- GFreeFunc data_free;
- gpointer data_user;
- };
-
- struct GstBufferVideoMeta {
- GstMeta meta
-
- GstBufferVideoFlags flags
-
- guint n_planes;
- GstVideoPlane plane[GST_VIDEO_MAX_PLANES];
- };
-
-
--- /dev/null
+GstMeta
+-------
+
+This document describes the design for arbitrary per-buffer metadata.
+
+Buffer metadata typically describes the lowlevel properties of the buffer
+content. These properties are typically not negotiated with caps but they are
+negotiated in the bufferpools.
+
+Some examples of metadata:
+
+ - pointers to buffer memory regions
+ - timestamp, duration
+ - offset, offset_end
+ - interlacing information
+ - video alignment, cropping, panning information
+ - extra container information such as granulepos, ...
+ - extra global buffer properties
+
+
+Requirements
+~~~~~~~~~~~~
+
+ - It must be fast
+ * allocation, free, low fragmentation
+ * access to the metadata fields, preferably not much slower than directly
+ accessing a C structure field
+ - It must be extensible. Elements should be able to add new arbitrary metadata
+ without requiring much effort. Also new metadata fields should not break API
+ or ABI.
+ - It plays nice with subbuffers. When a subbuffer is created, the various
+ buffer metadata should be copied/updated correctly.
+ - We should be able to negotiate metadata between elements
+
+Use cases
+---------
+
+ * DSP vs CPU caches
+
+ Both DSP and CPU can have separate MMUs and memory caches. When we exchange buffers
+ between two subsystems we need to flush caches so that one CPU can see the
+ modifications done by the other CPU. These caches must only be flushed when one
+ CPU performed a write and the other CPU needs to do a read.
+
+ In order to implement this we need to be able to mark our read and write
+ operations on the buffer data.
+
+ It might also be possible that buffers are not mapped into the address space of
+ the process normally and that an explicit mmap operation is needed to setup
+ the mapping tables for the physical memory.
+
+ * Video planes
+
+ Video data is sometimes allocated in non-contiguous planes for the Y and the UV
+ data. We need to be able to specify the data on a buffer using multiple
+ pointers in memory. We also need to be able to specify the stride for these
+ planes.
+
+ * Extra buffer data
+
+ Some elements might need to store extra data for a buffer. This is typically
+ done when the resources are allocated from another subsystem such as OMX or
+ X11.
+
+ * Processing information
+
+ Pan and crop information can be added to the buffer data when the downstream
+ element can understand and use this metadata. An imagesink can, for example,
+ use the pan and cropping formation when it blits the image on the screen
+ with little overhead.
+
+
+GstMeta
+~~~~~~~
+
+A GstMeta is a structure as follows:
+
+ struct _GstMeta {
+ GstMetaInfo *info; /* tag and info for the meta item */
+ };
+
+The purpose of the this structure is to serve as a common header for all metadata
+information that we can attach to a buffer. Specific metadata, such as timing metadata,
+will have this structure as the first field. For example:
+
+ struct _GstMetaTiming {
+ GstMeta meta; /* common meta header */
+
+ GstClockTime dts; /* decoding timestamp */
+ GstClockTime pts; /* presentation timestamp */
+ GstClockTime duration; /* duration of the data */
+ GstClockTime clock_rate; /* clock rate for the above values */
+ };
+
+Or another example for the buffer memory region that consists of some methods
+only.
+
+ struct _GstMetaMemory {
+ GstMeta meta;
+
+ GstMetaMemoryMap mmap_func;
+ GstMetaMemoryUnmap munmap_func;
+ };
+
+ typedef enum {
+ GST_BUFFER_MAP_NONE,
+ GST_BUFFER_MAP_READ,
+ GST_BUFFER_MAP_WRITE,
+ } GstBufferMapFlags
+
+ gpointer gst_meta_memory_map (GstMetaMemory *, guint offset, guint *size, GstBufferMapFlags);
+ gboolean gst_meta_memory_unmap (GstMetaMemory *, gpointer data, guint size);
+
+
+GstMeta derived structures define the API of the metadata. The API can consist of
+fields and/or methods. It is possible to have different implementations for the
+same GstMeta structure.
+
+The implementation of the GstMeta api would typically add more fields to the
+public structure that allow it to implement the API. For example:
+
+ struct _GstMetaMemoryImpl {
+ GstMetaMemory memory;
+
+ gpointer *data;
+ guint size;
+ gpointer *data_orig;
+ GFreeFunc data_free;
+ gpointer data_user;
+ };
+
+
+GstMetaInfo will point to more information about the metadata and looks like this:
+
+ struct _GstMetaInfo {
+ GQuark api; /* api name */
+ GQuark impl; /* implementation name */
+ gsize size; /* size of the structure */
+
+ GstMetaInitFunction init_func;
+ GstMetaFreeFunction free_func;
+ GstMetaTransformFunction transform_func;
+ GstMetaSerializeFunction serialize_func
+ GstMetaDeserializeFunction deserialize_func
+ };
+
+api will contain a GQuark of the metadata api. A repository of registered MetaInfo
+will be maintained by the core. We will register some common metadata structures
+in core and some media specific info for audio/video/text in -base. Plugins can
+register additional custom metadata.
+
+For each implementation of api, there will thus be a unique GstMetaInfo. In the
+case of metadata with a well defined API, the implementation specific init
+function will setup the methods in the metadata structure.
+
+Along with the metadata description we will have functions to initialize/free (and/or refcount)
+a specific GstMeta instance. We also have the possibility to add a custom
+transform function that can be used to modify the metadata when a transformation
+happens. Transformations can be copy, make-writable and subbuffer operations but
+can be expanded later.
+
+We also add serialize and deserialize function for the metadata in case we need special
+logic for reading and writing the metadata. This is needed for GDP payloading of the
+metadata.
+
+The purpose of the separate MetaInfo is to not have to carry the free/init functions in
+each buffer instance but to define them globally. We still want quick access to the info
+so we need to make the buffer metadata point to the info.
+
+Technically we could also specify the field and types in the MetaInfo and
+provide a generic API to retrieve the metadata fields without the need for a
+header file. We will not do this yet.
+
+Allocation of the GstBuffer structure will result in the allocation of a memory region
+of a customizable size (512 bytes). Only the first sizeof (GstBuffer) bytes of this
+region will initially be used. The remaining bytes will be part of the free metadata
+region of the buffer. Different implementations are possible and are invisible
+in the API or ABI.
+
+The complete buffer with metadata could, for example, look as follows:
+
+ +-------------------------------------+
+GstMiniObject | GType (GstBuffer) |
+ | refcount, flags, copy/disp/free |
+ +-------------------------------------+
+GstBuffer | caps, parent, pool |
+ +.....................................+
+ | next ---+
+ +- | info ------> GstMetaInfo
+GstMetaTiming | | | |
+ | | dts | |
+ | | pts | |
+ | | duration | |
+ +- | clock_rate | |
+ + . . . . . . . . . . . . . . . . . . + |
+ | next <--+
+GstMetaMemory +- +- | info ------> GstMetaInfo
+ | | | | |
+ | | | mmap | |
+ | | | munmap | |
+ +- | | | |
+ | | data | |
+GstMetaMemoryImpl | | size | |
+ | | mallocdata | |
+ | | data_free | |
+ +- | data_user | |
+ + . . . . . . . . . . . . . . . . . . + .
+ . .
+
+
+API examples
+~~~~~~~~~~~~
+
+Buffers are created using the normal gst_buffer_new functions. The standard fields
+are initialized as usual. A memory area that is bigger than the structure size
+is allocated for the buffer metadata.
+
+ gst_buffer_new ();
+
+After creating a buffer, the application can set caps and add metadata
+information.
+
+To add or retrieve metadata, a handle to a GstMetaInfo structure needs to be
+obtained. This defines the implementation and API of the metadata. Usually, a
+handle to this info structure can be obtained by calling a public _get_info()
+method from a shared library (for shared metadata).
+
+The following defines can usually be found in the shared .h file.
+
+ GstMetaInfo * gst_meta_timing_get_info();
+ #define GST_META_TIMING_INFO (gst_meta_timing_get_info())
+
+Adding metadata to a buffer can be done with the gst_buffer_add_meta() call.
+This function will create new metadata based on the implementation specified by
+the GstMetaInfo. It is alos possible to pass a generic pointer to the add_meta()
+function that can contain parameters to initialize the new metadata fields.
+
+Retrieving the metadata on a buffer can be done with the
+gst_buffer_meta_get() method. This function retrieves an existing metadata
+conforming to the API specified in the given info. When no such metadata exists,
+the function will return NULL.
+
+ GstMetaTiming *timing;
+
+ timing = gst_buffer_get_meta (buffer, GST_META_TIMING_INFO);
+
+Once a reference to the info has been obtained, the associated metadata can be
+added or modified on a buffer.
+
+ timing->timestamp = 0;
+ timing->duration = 20 * GST_MSECOND;
+
+Other convenience macros can be made to simplify the above code:
+
+ #define gst_buffer_get_meta_timing(b) \
+ ((GstMetaTiming *) gst_buffer_get_meta ((b), GST_META_TIMING_INFO)
+
+This makes the code look like this:
+
+ GstMetaTiming *timing;
+
+ timing = gst_buffer_get_meta_timing (buffer);
+ timing->timestamp = 0;
+ timing->duration = 20 * GST_MSECOND;
+
+To iterate the different metainfo structures, one can use the
+gst_buffer_meta_get_next() methods.
+
+ GstMeta *current = NULL;
+
+ /* passing NULL gives the first entry */
+ current = gst_buffer_meta_get_next (buffer, current);
+
+ /* passing a GstMeta returns the next */
+ current = gst_buffer_meta_get_next (buffer, current);
+
+
+Memory management
+~~~~~~~~~~~~~~~~~
+
+* allocation
+
+ We initially allocate a reasonable sized GstBuffer structure (say 512 bytes).
+
+ Since the complete buffer structure, including a large area for metadata, is
+ allocated in one go, we can reduce the number of memory allocations while still
+ providing dynamic metadata.
+
+ When adding metadata, we need to call the init function of the associated
+ metadata info structure. Since adding the metadata requires the caller to pass
+ a handle to the info, this operation does not require table lookups.
+
+ Per-metadata memory initialisation is needed because not all metadata is
+ initialized in the same way. We need to, for example, set the timestamps to
+ NONE in the MetaTiming structures.
+
+ The init/free functions can also be used to implement refcounting for a metadata
+ structure. This can be useful when a structure is shared between buffers.
+
+ When the free_size of the GstBuffer is exhausted, we will allocate new memory
+ for each newly added Meta and use the next pointers to point to this. It
+ is expected that this does not occur often and we might be able to optimize
+ this transparently in the future.
+
+* free
+
+ When a GstBuffer is freed, we potentially might have to call a custom free
+ function on the metadata info. In the case of the Memory metadata, we need to
+ call the associated free function to free the memory.
+
+ When freeing a GstBuffer, the custom buffer free function will iterate all of
+ the metadata in the buffer and call the associated free functions in the
+ MetaInfo associated with the entries. Usually, this function will be NULL.
+
+
+Serialization
+~~~~~~~~~~~~~
+
+When buffer should be sent over the wire or be serialized in GDP, we need a way
+to perform custom serialization and deserialization on the metadata.
+
+For this we add the serialize and deserialize functions to the metadata info.
+Possible use cases are to make sure we write out the fields with a specific size
+and endianness.
+
+
+Transformations
+~~~~~~~~~~~~~~~
+
+After certain transformations, the metadata on a buffer might not be relevant
+anymore.
+
+Consider, for example, metadata that lists certain regions of interest
+on the video data. If the video is scaled or rotated, the coordinates might not
+make sense anymore. A transform element should be able to adjust or remove the
+associated metadata when it becomes invalid.
+
+We can make the transform element aware of the metadata so that it can adjust or
+remove in an intelligent way. Since we allow arbitrary metadata, we can't do
+this for all metadata and thus we need some other way.
+
+One proposition is to tag the metadata type with keywords that specify what it
+functionally refers too. We could, for example, tag the metadata for the regions
+of interest with a tag that notes that the metadata refers to absolute pixel
+positions. A transform could then know that the metadata is not valid anymore
+when the position of the pixels changed (due to rotation, flipping, scaling and
+so on).
+
+
+Subbuffers
+~~~~~~~~~~
+
+Subbuffers are implemented with a generic transform. Parameters to the transform
+are the offset and size. This allows each metadata structure to implement the
+actions needed to update the metadata of the subbuffer.
+
+Since the subbuffer transform expects an offset and size, it might not make sense
+to make subbuffers from arbitrary buffers. Video metadata that has data in muliple
+planes, for example, might need to copy the planes to its 'natural' contiguous
+representation for the subbuffer or might simply ignore the subbuffer transform.
+
+
+Other use cases
+~~~~~~~~~~~~~~~
+
+Making the GstMetaMemory (for making the buffer point to the associated
+memory region) as metadata on a GstBuffer, as opposed to making it an integral
+part of GstBuffer, allows for some more interesting ways to transfer data.
+
+We could for example make a new GstMetaIOVec metadata structure like this:
+
+ struct _GstMetaIOVec {
+ GstMeta meta;
+
+ /* pointer to data and its size */
+ GFreeFunc data_free;
+ gpointer data_user;
+ guint len;
+ struct iovec *iov;
+ };
+
+This would allow us to transfer data in a scatter/gather array. Since the fields
+in the buffer metadata are now explicit, elements that don't support this kind
+of metadata can gracefully degrade.
+
+Another use case for not having the Memory metadata in the buffers would be for
+_pad_alloc() and get_range(). We can pass a GstBuffer with the requested
+metadata fields to those functions and have the _get_range() or pad_alloc()
+implementations add (or use, in the case of a file reader) the memory metadata.
+
+
+Relationship with GstCaps
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The difference between GstCaps, used in negotiation, and the metadata is not
+clearly defined.
+
+We would like to think of the GstCaps containing the information needed to
+functionally negotiate the format between two elements. The Metadata should then
+only contain variables that can change between each buffer.
+
+For example, for video we would have width/height/framerate in the caps but then
+have the more technical details, such as stride, data pointers, pan/crop/zoom
+etc in the metadata.
+
+A scheme like this would still allow us to functionally specify the desired
+video resolution while the implementation details would be inside the metadata.
+
+
+Compatibility
+~~~~~~~~~~~~~
+
+We need to make sure that elements exchange metadata that they both understand,
+This is particulary important when the metadata describes the data layout in
+memory (such as strides).
+
+We would like to use the bufferpool negotiation system to negotiate the possible
+metadata that can be exchanged between elements.
+
+When deciding the allocation properties, we will also negotiate the buffer
+metadata structures that we can exchange.
+
+
+Notes
+~~~~~
+
+Some structures that we need to be able to add to buffers.
+
+* Clean Aperture
+* Arbitrary Matrix Transform
+* Aspect ratio
+* Pan/crop/zoom
+* Video strides
+
+Some of these overlap, we need to find a minimal set of metadata structures that
+allows us to define all use cases.
+
+
+
+Video Buffers
+-------------
+
+ #define GST_VIDEO_MAX_PLANES 4
+
+ struct GstVideoPlane {
+ guint8 *data;
+ guint size;
+ guint stride;
+
+ guint8 *data_orig;
+ guint size_orig;
+ GFreeFunc data_free;
+ gpointer data_user;
+ };
+
+ struct GstBufferVideoMeta {
+ GstMeta meta
+
+ GstBufferVideoFlags flags
+
+ guint n_planes;
+ GstVideoPlane plane[GST_VIDEO_MAX_PLANES];
+ };
+
+
projects / platform / upstream / gstreamer.git / commitdiff