2 title: Memory allocation
7 Memory allocation and management is a very important topic in
8 multimedia. High definition video uses many megabytes to store one
9 single frame of video. It is important to reuse the memory when possible
10 instead of constantly allocating and freeing the memory.
12 Multimedia systems usually use special purpose chips, such as DSPs or
13 GPUs to perform the heavy lifting (especially for video). These special
14 purpose chips have usually strict requirements for the memory that they
15 can operate on and how the memory is accessed.
17 This chapter talks about the memory management features that GStreamer
18 plugins can use. We will first talk about the lowlevel `GstMemory`
19 object that manages access to a piece of memory. We then continue with
20 `GstBuffer` that is used to exchange data between plugins (and the
21 application) and that uses `GstMemory`. We talk about `GstMeta` that can
22 be placed on buffers to give extra info about the buffer and its memory.
23 For efficiently managing buffers of the same size, we take a look at
24 `GstBufferPool`. To conclude this chapter we take a look at the
25 GST\_QUERY\_ALLOCATION query that is used to negotiate memory management
26 options between elements.
30 `GstMemory` is an object that manages a region of memory. The memory
31 object points to a region of memory of “maxsize”. The area in this
32 memory starting at “offset” and for “size” bytes is the accessible
33 region in the memory. the maxsize of the memory can never be changed
34 after the object is created, however, the offset and size can be
39 `GstMemory` objects are created by a `GstAllocator` object. Most
40 allocators implement the default `gst_allocator_alloc()` method but some
41 allocator might implement a different method, for example when
42 additional parameters are needed to allocate the specific memory.
44 Different allocators exist for, for example, system memory, shared
45 memory and memory backed by a DMAbuf file descriptor. To implement
46 support for a new kind of memory type, you must implement a new
47 allocator object as shown below.
49 ### GstMemory API example
51 Data access to the memory wrapped by the `GstMemory` object is always
52 protected with a `gst_memory_map()` and `gst_memory_unmap()` pair. An
53 access mode (read/write) must be given when mapping memory. The map
54 function returns a pointer to the valid memory region that can then be
55 accessed according to the requested access mode.
57 Below is an example of making a `GstMemory` object and using the
58 `gst_memory_map()` to access the memory region.
68 /* allocate 100 bytes */
69 mem = gst_allocator_alloc (NULL, 100, NULL);
71 /* get access to the memory in write mode */
72 gst_memory_map (mem, &info, GST_MAP_WRITE);
74 /* fill with pattern */
75 for (i = 0; i < info.size; i++)
79 gst_memory_unmap (mem, &info);
86 ### Implementing a GstAllocator
92 A `GstBuffer` is an lightweight object that is passed from an upstream
93 to a downstream element and contains memory and metadata. It represents
94 the multimedia content that is pushed or pull downstream by elements.
96 The buffer contains one or more `GstMemory` objects that represent the
99 Metadata in the buffer consists of:
101 - DTS and PTS timestamps. These represent the decoding and
102 presentation timestamps of the buffer content and is used by
103 synchronizing elements to schedule buffers. Both these timestamps
104 can be GST\_CLOCK\_TIME\_NONE when unknown/undefined.
106 - The duration of the buffer contents. This duration can be
107 GST\_CLOCK\_TIME\_NONE when unknown/undefined.
109 - Media specific offsets and offset\_end. For video this is the frame
110 number in the stream and for audio the sample number. Other
111 definitions for other media exist.
113 - Arbitrary structures via `GstMeta`, see below.
115 ### GstBuffer writability
117 A buffer is writable when the refcount of the object is exactly 1,
118 meaning that only one object is holding a ref to the buffer. You can
119 only modify anything in the buffer when the buffer is writable. This
120 means that you need to call `gst_buffer_make_writable()` before changing
121 the timestamps, offsets, metadata or adding and removing memory blocks.
123 ### GstBuffer API examples
125 You can create a buffer with `gst_buffer_new ()` and then add memory
126 objects to it or you can use a convenience function
127 `gst_buffer_new_allocate ()` which combines the two. It's also possible
128 to wrap existing memory with `gst_buffer_new_wrapped_full () ` where you
129 can give the function to call when the memory should be freed.
131 You can access the memory of the buffer by getting and mapping the
132 `GstMemory` objects individually or by using `gst_buffer_map ()`. The
133 latter merges all the memory into one big block and then gives you a
134 pointer to this block.
136 Below is an example of how to create a buffer and access its memory.
145 /* make empty buffer */
146 buffer = gst_buffer_new ();
148 /* make memory holding 100 bytes */
149 mem = gst_allocator_alloc (NULL, 100, NULL);
152 gst_buffer_append_memory (buffer, mem);
156 /* get WRITE access to the memory and fill with 0xff */
157 gst_buffer_map (buffer, &info, GST_MAP_WRITE);
158 memset (info.data, 0xff, info.size);
159 gst_buffer_unmap (buffer, &info);
163 /* free the buffer */
164 gst_buffer_unref (buffer);
173 With the `GstMeta` system you can add arbitrary structures on buffers.
174 These structures describe extra properties of the buffer such as
175 cropping, stride, region of interest etc.
177 The metadata system separates API specification (what the metadata and
178 its API look like) and the implementation (how it works). This makes it
179 possible to make different implementations of the same API, for example,
180 depending on the hardware you are running on.
182 ### GstMeta API example
184 After allocating a new buffer, you can add metadata to the buffer with
185 the metadata specific API. This means that you will need to link to the
186 header file where the metadata is defined to use its API.
188 By convention, a metadata API with name `FooBar` should provide two
189 methods, a `gst_buffer_add_foo_bar_meta ()` and a
190 `gst_buffer_get_foo_bar_meta ()`. Both functions should return a pointer
191 to a `FooBarMeta` structure that contains the metadata fields. Some of
192 the `_add_*_meta ()` can have extra parameters that will usually be used
193 to configure the metadata structure for you.
195 Let's have a look at the metadata that is used to specify a cropping
196 region for video frames.
200 #include <gst/video/gstvideometa.h>
203 GstVideoCropMeta *meta;
205 /* buffer points to a video frame, add some cropping metadata */
206 meta = gst_buffer_add_video_crop_meta (buffer);
208 /* configure the cropping metadata */
218 An element can then use the metadata on the buffer when rendering the
223 #include <gst/video/gstvideometa.h>
226 GstVideoCropMeta *meta;
228 /* buffer points to a video frame, get the cropping metadata */
229 meta = gst_buffer_get_video_crop_meta (buffer);
232 /* render frame with cropping */
233 _render_frame_cropped (buffer, meta->x, meta->y, meta->width, meta->height);
236 _render_frame (buffer);
244 ### Implementing new GstMeta
246 In the next sections we show how you can add new metadata to the system
247 and use it on buffers.
249 #### Define the metadata API
251 First we need to define what our API will look like and we will have to
252 register this API to the system. This is important because this API
253 definition will be used when elements negotiate what kind of metadata
254 they will exchange. The API definition also contains arbitrary tags that
255 give hints about what the metadata contains. This is important when we
256 see how metadata is preserved when buffers pass through the pipeline.
258 If you are making a new implementation of an existing API, you can skip
259 this step and move on to the implementation step.
261 First we start with making the `my-example-meta.h` header file that will
262 contain the definition of the API and structure for our metadata.
268 typedef struct _MyExampleMeta MyExampleMeta;
270 struct _MyExampleMeta {
277 GType my_example_meta_api_get_type (void);
278 #define MY_EXAMPLE_META_API_TYPE (my_example_meta_api_get_type())
280 #define gst_buffer_get_my_example_meta(b) \
281 ((MyExampleMeta*)gst_buffer_get_meta((b),MY_EXAMPLE_META_API_TYPE))
286 The metadata API definition consists of the definition of the structure
287 that holds a gint and a string. The first field in the structure must be
290 We also define a `my_example_meta_api_get_type ()` function that will
291 register out metadata API definition. We also define a convenience macro
292 `gst_buffer_get_my_example_meta ()` that simply finds and returns the
293 metadata with our new API.
295 Next let's have a look at how the `my_example_meta_api_get_type ()`
296 function is implemented in the `my-example-meta.c` file.
300 #include "my-example-meta.h"
303 my_example_meta_api_get_type (void)
305 static volatile GType type;
306 static const gchar *tags[] = { "foo", "bar", NULL };
308 if (g_once_init_enter (&type)) {
309 GType _type = gst_meta_api_type_register ("MyExampleMetaAPI", tags);
310 g_once_init_leave (&type, _type);
318 As you can see, it simply uses the `gst_meta_api_type_register ()`
319 function to register a name for the api and some tags. The result is a
320 new pointer GType that defines the newly registered API.
322 #### Implementing a metadata API
324 Next we can make an implementation for a registered metadata API GType.
325 The implementation detail of a metadata API are kept in a `GstMetaInfo`
326 structure that you will make available to the users of your metadata API
327 implementation with a `my_example_meta_get_info ()` function and a
328 convenience `MY_EXAMPLE_META_INFO` macro. You will also make a method to
329 add your metadata implementation to a `GstBuffer`. Your
330 `my-example-meta.h` header file will need these additions:
337 const GstMetaInfo *my_example_meta_get_info (void);
338 #define MY_EXAMPLE_META_INFO (my_example_meta_get_info())
340 MyExampleMeta * gst_buffer_add_my_example_meta (GstBuffer *buffer,
347 Let's have a look at how these functions are implemented in the
348 `my-example-meta.c` file.
355 my_example_meta_init (GstMeta * meta, gpointer params, GstBuffer * buffer)
357 MyExampleMeta *emeta = (MyExampleMeta *) meta;
366 my_example_meta_transform (GstBuffer * transbuf, GstMeta * meta,
367 GstBuffer * buffer, GQuark type, gpointer data)
369 MyExampleMeta *emeta = (MyExampleMeta *) meta;
371 /* we always copy no matter what transform */
372 gst_buffer_add_my_example_meta (transbuf, emeta->age, emeta->name);
378 my_example_meta_free (GstMeta * meta, GstBuffer * buffer)
380 MyExampleMeta *emeta = (MyExampleMeta *) meta;
382 g_free (emeta->name);
387 my_example_meta_get_info (void)
389 static const GstMetaInfo *meta_info = NULL;
391 if (g_once_init_enter (&meta_info)) {
392 const GstMetaInfo *mi = gst_meta_register (MY_EXAMPLE_META_API_TYPE,
394 sizeof (MyExampleMeta),
395 my_example_meta_init,
396 my_example_meta_free,
397 my_example_meta_transform);
398 g_once_init_leave (&meta_info, mi);
404 gst_buffer_add_my_example_meta (GstBuffer *buffer,
410 g_return_val_if_fail (GST_IS_BUFFER (buffer), NULL);
412 meta = (MyExampleMeta *) gst_buffer_add_meta (buffer,
413 MY_EXAMPLE_META_INFO, NULL);
416 meta->name = g_strdup (name);
424 `gst_meta_register ()` registers the implementation details, like the
425 API that you implement and the size of the metadata structure along with
426 methods to initialize and free the memory area. You can also implement a
427 transform function that will be called when a certain transformation
428 (identified by the quark and quark specific data) is performed on a
431 Lastly, you implement a `gst_buffer_add_*_meta()` that adds the metadata
432 implementation to a buffer and sets the values of the metadata.
436 The `GstBufferPool` object provides a convenient base class for managing
437 lists of reusable buffers. Essential for this object is that all the
438 buffers have the same properties such as size, padding, metadata and
441 A bufferpool object can be configured to manage a minimum and maximum
442 amount of buffers of a specific size. A bufferpool can also be
443 configured to use a specific `GstAllocator` for the memory of the
444 buffers. There is support in the bufferpool to enable bufferpool
445 specific options, such as adding `GstMeta` to the buffers in the pool or
446 such as enabling specific padding on the memory in the buffers.
448 A Bufferpool can be inactivate and active. In the inactive state, you
449 can configure the pool. In the active state, you can't change the
450 configuration anymore but you can acquire and release buffers from/to
453 In the following sections we take a look at how you can use a
456 ### GstBufferPool API example
458 Many different bufferpool implementations can exist; they are all
459 subclasses of the base class `GstBufferPool`. For this example, we will
460 assume we somehow have access to a bufferpool, either because we created
461 it ourselves or because we were given one as a result of the ALLOCATION
462 query as we will see below.
464 The bufferpool is initially in the inactive state so that we can
465 configure it. Trying to configure a bufferpool that is not in the
466 inactive state will fail. Likewise, trying to activate a bufferpool that
467 is not configured will fail.
471 GstStructure *config;
475 /* get config structure */
476 config = gst_buffer_pool_get_config (pool);
478 /* set caps, size, minimum and maximum buffers in the pool */
479 gst_buffer_pool_config_set_params (config, caps, size, min, max);
481 /* configure allocator and parameters */
482 gst_buffer_pool_config_set_allocator (config, allocator, ¶ms);
484 /* store the updated configuration again */
485 gst_buffer_pool_set_config (pool, config);
492 The configuration of the bufferpool is maintained in a generic
493 `GstStructure` that can be obtained with `gst_buffer_pool_get_config()`.
494 Convenience methods exist to get and set the configuration options in
495 this structure. After updating the structure, it is set as the current
496 configuration in the bufferpool again with
497 `gst_buffer_pool_set_config()`.
499 The following options can be configured on a bufferpool:
501 - The caps of the buffers to allocate.
503 - The size of the buffers. This is the suggested size of the buffers
504 in the pool. The pool might decide to allocate larger buffers to add
507 - The minimum and maximum amount of buffers in the pool. When minimum
508 is set to \> 0, the bufferpool will pre-allocate this amount of
509 buffers. When maximum is not 0, the bufferpool will allocate up to
510 maximum amount of buffers.
512 - The allocator and parameters to use. Some bufferpools might ignore
513 the allocator and use its internal one.
515 - Other arbitrary bufferpool options identified with a string. a
516 bufferpool lists the supported options with
517 `gst_buffer_pool_get_options()` and you can ask if an option is
518 supported with `gst_buffer_pool_has_option()`. The option can be
519 enabled by adding it to the configuration structure with
520 `gst_buffer_pool_config_add_option ()`. These options are used to
521 enable things like letting the pool set metadata on the buffers or
522 to add extra configuration options for padding, for example.
524 After the configuration is set on the bufferpool, the pool can be
525 activated with `gst_buffer_pool_set_active (pool, TRUE)`. From that
526 point on you can use `gst_buffer_pool_acquire_buffer ()` to retrieve a
527 buffer from the pool, like this:
536 ret = gst_buffer_pool_acquire_buffer (pool, &buffer, NULL);
537 if (G_UNLIKELY (ret != GST_FLOW_OK))
545 It is important to check the return value of the acquire function
546 because it is possible that it fails: When your element shuts down, it
547 will deactivate the bufferpool and then all calls to acquire will return
550 All buffers that are acquired from the pool will have their pool member
551 set to the original pool. When the last ref is decremented on the
552 buffer, GStreamer will automatically call
553 `gst_buffer_pool_release_buffer()` to release the buffer back to the
554 pool. You (or any other downstream element) don't need to know if a
555 buffer came from a pool, you can just unref it.
557 ### Implementing a new GstBufferPool
561 ## GST\_QUERY\_ALLOCATION
563 The ALLOCATION query is used to negotiate `GstMeta`, `GstBufferPool` and
564 `GstAllocator` between elements. Negotiation of the allocation strategy
565 is always initiated and decided by a srcpad after it has negotiated a
566 format and before it decides to push buffers. A sinkpad can suggest an
567 allocation strategy but it is ultimately the source pad that will decide
568 based on the suggestions of the downstream sink pad.
570 The source pad will do a GST\_QUERY\_ALLOCATION with the negotiated caps
571 as a parameter. This is needed so that the downstream element knows what
572 media type is being handled. A downstream sink pad can answer the
573 allocation query with the following results:
575 - An array of possible `GstBufferPool` suggestions with suggested
576 size, minimum and maximum amount of buffers.
578 - An array of GstAllocator objects along with suggested allocation
579 parameters such as flags, prefix, alignment and padding. These
580 allocators can also be configured in a bufferpool when this is
581 supported by the bufferpool.
583 - An array of supported `GstMeta` implementations along with metadata
584 specific parameters. It is important that the upstream element knows
585 what kind of metadata is supported downstream before it places that
588 When the GST\_QUERY\_ALLOCATION returns, the source pad will select from
589 the available bufferpools, allocators and metadata how it will allocate
592 ### ALLOCATION query example
594 Below is an example of the ALLOCATION query.
598 #include <gst/video/video.h>
599 #include <gst/video/gstvideometa.h>
600 #include <gst/video/gstvideopool.h>
604 GstStructure *structure;
606 GstStructure *config;
607 guint size, min, max;
611 /* find a pool for the negotiated caps now */
612 query = gst_query_new_allocation (caps, TRUE);
614 if (!gst_pad_peer_query (scope->srcpad, query)) {
615 /* query failed, not a problem, we use the query defaults */
618 if (gst_query_get_n_allocation_pools (query) > 0) {
619 /* we got configuration from our peer, parse them */
620 gst_query_parse_nth_allocation_pool (query, 0, &pool, &size, &min, &max);
628 /* we did not get a pool, make one ourselves then */
629 pool = gst_video_buffer_pool_new ();
632 config = gst_buffer_pool_get_config (pool);
633 gst_buffer_pool_config_add_option (config, GST_BUFFER_POOL_OPTION_VIDEO_META);
634 gst_buffer_pool_config_set_params (config, caps, size, min, max);
635 gst_buffer_pool_set_config (pool, config);
638 gst_buffer_pool_set_active (pool, TRUE);
645 This particular implementation will make a custom `GstVideoBufferPool`
646 object that is specialized in allocating video buffers. You can also
647 enable the pool to put `GstVideoMeta` metadata on the buffers from the
648 pool doing `gst_buffer_pool_config_add_option (config,
649 GST_BUFFER_POOL_OPTION_VIDEO_META)`.
651 ### The ALLOCATION query in base classes
653 In many baseclasses you will see the following virtual methods for
654 influencing the allocation strategy:
656 - `propose_allocation ()` should suggest allocation parameters for the
659 - `decide_allocation ()` should decide the allocation parameters from
660 the suggestions received from downstream.
662 Implementors of these methods should modify the given `GstQuery` object
663 by updating the pool options and allocation options.