1 <chapter id="chapter-allocation" xreflabel="Memory allocation">
2 <title>Memory allocation</title>
4 Memory allocation and management is a very important topic in
5 multimedia. High definition video uses many magabytes to store
6 one single frame of video. It is important to reuse the memory
7 when possible instead of constantly allocating and freeing
11 Multimedia systems usually use special purpose chips, such as
12 DSPs or GPUs to perform the heavy lifting (especially for video).
13 These special purpose chips have usually strict requirements
14 for the memory that they can operate on and how the memory
18 This chapter talks about the memory management features that
19 &GStreamer; plugins can use. We will first talk about the
20 lowlevel <classname>GstMemory</classname> object that manages
21 access to a piece of memory. We then continue with
22 <classname>GstBuffer</classname> that is used to exchange data
23 between plugins (and the application) and that uses
24 <classname>GstMemory</classname>. We talk about
25 <classname>GstMeta</classname> that can be placed on buffers to
26 give extra info about the buffer and its memory.
27 For efficiently managing buffers of the same size, we take a
28 look at <classname>GstBufferPool</classname>. To conclude this
29 chapter we take a look at the GST_QUERY_ALLOCATION query that
30 is used to negotiate memory management options between elements.
33 <sect1 id="section-allocation-memory" xreflabel="GstMemory">
34 <title>GstMemory</title>
36 <classname>GstMemory</classname> is an object that manages a region
37 of memory. The memory object points to a region of memory of
38 <quote>maxsize</quote>. The area in this memory starting at
39 <quote>offset</quote> and for <quote>size</quote> bytes is the
40 accessible region in the memory. the maxsize of the memory can
41 never be changed after the object is created, however, the offset
42 and size can be changed.
44 <sect2 id="section-allocation-allocator" xreflabel="GstAllocator">
45 <title>GstAllocator</title>
47 <classname>GstMemory</classname> objects are created by a
48 <classname>GstAllocator</classname> object. Most allocators implement the
49 default <function>gst_allocator_alloc()</function> method but some allocator
50 might implement a different method, for example when additional parameters
51 are needed to allocate the specific memory.
54 Different allocators exist for, for example, system memory, shared memory
55 and memory backed by a DMAbuf file descriptor. To implement support for a
56 new kind of memory type, you must implement a new allocator object as shown
60 <sect2 id="section-allocation-memory-ex" xreflabel="GstMemory-ex">
61 <title>GstMemory API example</title>
63 Data access to the memory wrapped by the <classname>GstMemory</classname>
64 object is always protected with a <function>gst_memory_map()</function>
65 and <function>gst_memory_unmap()</function> pair. An access mode
66 (read/write) must be given when mapping memory. The map
67 function returns a pointer to the valid memory region that can
68 then be accessed according to the requested access mode.
71 Below is an example of making a <classname>GstMemory</classname>
72 object and using the <function>gst_memory_map()</function> to
73 access the memory region.
83 /* allocate 100 bytes */
84 mem = gst_allocator_alloc (NULL, 100, NULL);
86 /* get access to the memory in write mode */
87 gst_memory_map (mem, &info, GST_MAP_WRITE);
89 /* fill with pattern */
90 for (i = 0; i < info.size; i++)
94 gst_memory_unmap (mem, &info);
101 <sect2 id="section-allocation-allocator-ex" xreflabel="GstAllocator-ex">
102 <title>Implementing a GstAllocator</title>
110 <sect1 id="section-allocation-buffer" xreflabel="GstBuffer">
111 <title>GstBuffer</title>
113 A <classname>GstBuffer</classname> is an lightweight object that
114 is passed from an upstream to a downstream element and contains
115 memory and metadata. It represents the multimedia content that
116 is pushed or pull downstream by elements.
119 The buffer contains one or more <classname>GstMemory</classname>
120 objects thet represent the data in the buffer.
123 Metadata in the buffer consists of:
125 <itemizedlist mark="opencircle">
128 DTS and PTS timestamps. These represent the decoding and
129 presentation timestamps of the buffer content and is used by
130 synchronizing elements to schedule buffers. Both these timestamps
131 can be GST_CLOCK_TIME_NONE when unknown/undefined.
136 The duration of the buffer contents. This duration can be
137 GST_CLOCK_TIME_NONE when unknown/undefined.
142 Media specific offsets and offset_end. For video this is the
143 frame number in the stream and for audio the sample number. Other
144 definitions for other media exist.
149 Arbitrary structures via <classname>GstMeta</classname>, see below.
154 <sect2 id="section-allocation-writability" xreflabel="GstBuffer-write">
155 <title>GstBuffer writability</title>
157 A buffer is writable when the refcount of the object is exactly 1, meaning
158 that only one object is holding a ref to the buffer. You can only
159 modify anything in the buffer when the buffer is writable. This means
160 that you need to call <function>gst_buffer_make_writable()</function>
161 before changing the timestamps, offsets, metadata or adding and
162 removing memory blocks.
165 <sect2 id="section-allocation-buffer-ex" xreflabel="GstBuffer-ex">
166 <title>GstBuffer API examples</title>
168 You can create a buffer with <function>gst_buffer_new ()</function>
169 and then add memory objects to it or you can use a convenience function
170 <function>gst_buffer_new_allocate ()</function> which combines the
171 two. It's also possible to wrap existing memory with
172 <function>gst_buffer_new_wrapped_full () </function> where you can
173 give the function to call when the memory should be freed.
176 You can access the memory of the buffer by getting and mapping the
177 <classname>GstMemory</classname> objects individually or by using
178 <function>gst_buffer_map ()</function>. The latter merges all the
179 memory into one big block and then gives you a pointer to this block.
182 Below is an example of how to create a buffer and access its memory.
191 /* make empty buffer */
192 buffer = gst_buffer_new ();
194 /* make memory holding 100 bytes */
195 mem = gst_allocator_alloc (NULL, 100, NULL);
197 /* add the the buffer */
198 gst_buffer_append_memory (buffer, mem);
202 /* get WRITE access to the memory and fill with 0xff */
203 gst_buffer_map (buffer, &info, GST_MAP_WRITE);
204 memset (info.data, 0xff, info.size);
205 gst_buffer_unmap (buffer, &info);
209 /* free the buffer */
210 gst_buffer_unref (buffer);
218 <sect1 id="section-allocation-meta" xreflabel="GstMeta">
219 <title>GstMeta</title>
221 With the <classname>GstMeta</classname> system you can add arbitrary
222 structures on buffers. These structures describe extra properties
223 of the buffer such as cropping, stride, region of interest etc.
226 The metadata system separates API specification (what the metadata
227 and its API look like) and the implementation (how it works). This makes
228 it possible to make different implementations of the same API,
229 for example, depending on the hardware you are running on.
232 <sect2 id="section-allocation-meta-ex" xreflabel="GstMeta-ex">
233 <title>GstMeta API example</title>
235 After allocating a new buffer, you can add metadata to the buffer
236 with the metadata specific API. This means that you will need to
237 link to the header file where the metadata is defined to use
241 By convention, a metadata API with name <classname>FooBar</classname>
242 should provide two methods, a
243 <function>gst_buffer_add_foo_bar_meta ()</function> and a
244 <function>gst_buffer_get_foo_bar_meta ()</function>. Both functions
245 should return a pointer to a <classname>FooBarMeta</classname>
246 structure that contains the metadata fields. Some of the
247 <function>_add_*_meta ()</function> can have extra parameters that
248 will usually be used to configure the metadata structure for you.
251 Let's have a look at the metadata that is used to specify a cropping
252 region for video frames.
256 #include <gst/video/gstvideometa.h>
259 GstVideoCropMeta *meta;
261 /* buffer points to a video frame, add some cropping metadata */
262 meta = gst_buffer_add_video_crop_meta (buffer);
264 /* configure the cropping metadata */
273 An element can then use the metadata on the buffer when rendering
278 #include <gst/video/gstvideometa.h>
281 GstVideoCropMeta *meta;
283 /* buffer points to a video frame, get the cropping metadata */
284 meta = gst_buffer_get_video_crop_meta (buffer);
287 /* render frame with cropping */
288 _render_frame_cropped (buffer, meta->x, meta->y, meta->width, meta->height);
291 _render_frame (buffer);
299 <sect2 id="section-allocation-meta-new" xreflabel="GstMeta-new">
300 <title>Implementing new GstMeta</title>
302 In the next sections we show how you can add new metadata to the
303 system and use it on buffers.
306 <sect3 id="section-allocation-meta-api" xreflabel="GstMeta-api">
307 <title>Define the metadata API</title>
309 First we need to define what our API will look like and we
310 will have to register this API to the system. This is important
311 because this API definition will be used when elements negotiate
312 what kind of metadata they will exchange. The API definition
313 also contains arbitrary tags that give hints about what the
314 metadata contains. This is important when we see how metadata
315 is preserved when buffers pass through the pipeline.
318 If you are making a new implementation of an existing API,
319 you can skip this step and move on to the implementation step.
322 First we start with making the
323 <filename>my-example-meta.h</filename> header file that will contain
324 the definition of the API and structure for our metadata.
330 typedef struct _MyExampleMeta MyExampleMeta;
332 struct _MyExampleMeta {
339 GType my_example_meta_api_get_type (void);
340 #define MY_EXAMPLE_META_API_TYPE (my_example_meta_api_get_type())
342 #define gst_buffer_get_my_example_meta(b) \
343 ((MyExampleMeta*)gst_buffer_get_meta((b),MY_EXAMPLE_META_API_TYPE))
347 The metadata API definition consists of the definition of the
348 structure that holds a gint and a string. The first field in
349 the structure must be <classname>GstMeta</classname>.
352 We also define a <function>my_example_meta_api_get_type ()</function>
353 function that will register out metadata API definition. We
354 also define a convenience macro
355 <function>gst_buffer_get_my_example_meta ()</function> that simply
356 finds and returns the metadata with our new API.
359 Next let's have a look at how the
360 <function>my_example_meta_api_get_type ()</function> function is
361 implemented in the <filename>my-example-meta.c</filename> file.
365 #include "my-example-meta.h"
368 my_example_meta_api_get_type (void)
370 static volatile GType type;
371 static const gchar *tags[] = { "foo", "bar", NULL };
373 if (g_once_init_enter (&type)) {
374 GType _type = gst_meta_api_type_register ("MyExampleMetaAPI", tags);
375 g_once_init_leave (&type, _type);
382 As you can see, it simply uses the
383 <function>gst_meta_api_type_register ()</function> function to
384 register a name for the api and some tags. The result is a
385 new pointer GType that defines the newly registered API.
389 <sect3 id="section-allocation-meta-impl" xreflabel="GstMeta-impl">
390 <title>Implementing a metadata API</title>
392 Next we can make an implementation for a registered metadata
393 API GType. The implementation detail of a metadata API
394 are kept in a <classname>GstMetaInfo</classname> structure
395 that you will make available to the users of your metadata
396 API implementation with a <function>my_example_meta_get_info ()</function>
397 function and a convenience <function>MY_EXAMPLE_META_INFO</function>
398 macro. You will also make a method to add your metadata
399 implementation to a <classname>GstBuffer</classname>.
400 Your <filename>my-example-meta.h</filename> header file will
408 const GstMetaInfo *my_example_meta_get_info (void);
409 #define MY_EXAMPLE_META_INFO (my_example_meta_get_info())
411 MyExampleMeta * gst_buffer_add_my_example_meta (GstBuffer *buffer,
417 Let's have a look at how these functions are
418 implemented in the <filename>my-example-meta.c</filename> file.
425 my_example_meta_init (GstMeta * meta, gpointer params, GstBuffer * buffer)
427 MyExampleMeta *emeta = (MyExampleMeta *) meta;
436 my_example_meta_transform (GstBuffer * transbuf, GstMeta * meta,
437 GstBuffer * buffer, GQuark type, gpointer data)
439 MyExampleMeta *emeta = (MyExampleMeta *) meta;
441 /* we always copy no matter what transform */
442 gst_buffer_add_my_example_meta (transbuf, emeta->age, emeta->name);
448 my_example_meta_free (GstMeta * meta, GstBuffer * buffer)
450 MyExampleMeta *emeta = (MyExampleMeta *) meta;
457 my_example_meta_get_info (void)
459 static const GstMetaInfo *meta_info = NULL;
461 if (g_once_init_enter (&meta_info)) {
462 const GstMetaInfo *mi = gst_meta_register (MY_EXAMPLE_META_API_TYPE,
464 sizeof (MyExampleMeta),
465 my_example_meta_init,
466 my_example_meta_free,
467 my_example_meta_transform);
468 g_once_init_leave (&meta_info, mi);
474 gst_buffer_add_my_example_meta (GstBuffer *buffer,
480 g_return_val_if_fail (GST_IS_BUFFER (buffer), NULL);
482 meta = (MyExampleMeta *) gst_buffer_add_meta (buffer,
483 MY_EXAMPLE_META_INFO, NULL);
486 meta->name = g_strdup (name);
493 <function>gst_meta_register ()</function> registers the implementation
494 details, like the API that you implement and the size of the
495 metadata structure along with methods to initialize and free the
496 memory area. You can also implement a transform function that will
497 be called when a certain transformation (identified by the quark and
498 quark specific data) is performed on a buffer.
501 Lastly, you implement a <function>gst_buffer_add_*_meta()</function>
502 that adds the metadata implementation to a buffer and sets the
503 values of the metadata.
510 <sect1 id="section-allocation-bufferpool" xreflabel="GstBufferPool">
511 <title>GstBufferPool</title>
513 The <classname>GstBufferPool</classname> object provides a convenient
514 base class for managing lists of reusable buffers. Essential for this
515 object is that all the buffers have the same properties such as size,
516 padding, metadata and alignment.
519 A bufferpool object can be configured to manage a minimum and maximum
520 amount of buffers of a specific size. A bufferpool can also be
521 configured to use a specific <classname>GstAllocator</classname> for
522 the memory of the buffers. There is support in the bufferpool to enable
523 bufferpool specific options, such as adding <classname>GstMeta</classname>
524 to the buffers in the pool or such as enabling specific padding on
525 the memory in the buffers.
528 A Bufferpool can be inactivate and active. In the inactive state,
529 you can configure the pool. In the active state, you can't change
530 the configuration anymore but you can acquire and release buffers
534 In the following sections we take a look at how you can use
538 <sect2 id="section-allocation-pool-ex" xreflabel="GstBufferPool-ex">
539 <title>GstBufferPool API example</title>
541 Many different bufferpool implementations can exist; they are all
542 subclasses of the base class <classname>GstBufferPool</classname>.
543 For this example, we will assume we somehow have access to a
544 bufferpool, either because we created it ourselves or because
545 we were given one as a result of the ALLOCATION query as we will
549 The bufferpool is initially in the inactive state so that we can
550 configure it. Trying to configure a bufferpool that is not in the
551 inactive state will fail. Likewise, trying to activate a bufferpool
552 that is not configured will fail.
556 GstStructure *config;
560 /* get config structure */
561 config = gst_buffer_pool_get_config (pool);
563 /* set caps, size, minimum and maximum buffers in the pool */
564 gst_buffer_pool_config_set_params (config, caps, size, min, max);
566 /* configure allocator and parameters */
567 gst_buffer_pool_config_set_allocator (config, allocator, ¶ms);
569 /* store the updated configuration again */
570 gst_buffer_pool_set_config (pool, config);
576 The configuration of the bufferpool is maintained in a generic
577 <classname>GstStructure</classname> that can be obtained with
578 <function>gst_buffer_pool_get_config()</function>. Convenience
579 methods exist to get and set the configuration options in this
580 structure. After updating the structure, it is set as the current
581 configuration in the bufferpool again with
582 <function>gst_buffer_pool_set_config()</function>.
585 The following options can be configured on a bufferpool:
587 <itemizedlist mark="opencircle">
590 The caps of the buffers to allocate.
595 The size of the buffers. This is the suggested size of the
596 buffers in the pool. The pool might decide to allocate larger
597 buffers to add padding.
602 The minimum and maximum amount of buffers in the pool. When
603 minimum is set to > 0, the bufferpool will pre-allocate this
604 amount of buffers. When maximum is not 0, the bufferpool
605 will allocate up to maximum amount of buffers.
610 The allocator and parameters to use. Some bufferpools might
611 ignore the allocator and use its internal one.
616 Other arbitrary bufferpool options identified with a string.
617 a bufferpool lists the supported options with
618 <function>gst_buffer_pool_get_options()</function> and you
619 can ask if an option is supported with
620 <function>gst_buffer_pool_has_option()</function>. The option
621 can be enabled by adding it to the configuration structure
622 with <function>gst_buffer_pool_config_add_option ()</function>.
623 These options are used to enable things like letting the
624 pool set metadata on the buffers or to add extra configuration
625 options for padding, for example.
630 After the configuration is set on the bufferpool, the pool can
632 <function>gst_buffer_pool_set_active (pool, TRUE)</function>. From
633 that point on you can use
634 <function>gst_buffer_pool_acquire_buffer ()</function> to retrieve
635 a buffer from the pool, like this:
644 ret = gst_buffer_pool_acquire_buffer (pool, &buffer, NULL);
645 if (G_UNLIKELY (ret != GST_FLOW_OK))
652 It is important to check the return value of the acquire function
653 because it is possible that it fails: When your
654 element shuts down, it will deactivate the bufferpool and then
655 all calls to acquire will return GST_FLOW_FLUSHNG.
658 All buffers that are acquired from the pool will have their pool
659 member set to the original pool. When the last ref is decremented
660 on the buffer, &GStreamer; will automatically call
661 <function>gst_buffer_pool_release_buffer()</function> to release
662 the buffer back to the pool. You (or any other downstream element)
663 don't need to know if a buffer came from a pool, you can just
668 <sect2 id="section-allocation-pool-impl" xreflabel="GstBufferPool-impl">
669 <title>Implementing a new GstBufferPool</title>
677 <sect1 id="section-allocation-query" xreflabel="GST_QUERY_ALLOCATION">
678 <title>GST_QUERY_ALLOCATION</title>
680 The ALLOCATION query is used to negotiate
681 <classname>GstMeta</classname>, <classname>GstBufferPool</classname>
682 and <classname>GstAllocator</classname> between elements. Negotiation
683 of the allocation strategy is always initiated and decided by a srcpad
684 after it has negotiated a format and before it decides to push buffers.
685 A sinkpad can suggest an allocation strategy but it is ultimately the
686 source pad that will decide based on the suggestions of the downstream
690 The source pad will do a GST_QUERY_ALLOCATION with the negotiated caps
691 as a parameter. This is needed so that the downstream element knows
692 what media type is being handled. A downstream sink pad can answer the
693 allocation query with the following results:
695 <itemizedlist mark="opencircle">
698 An array of possible <classname>GstBufferPool</classname> suggestions
699 with suggested size, minimum and maximum amount of buffers.
704 An array of GstAllocator objects along with suggested allocation
705 parameters such as flags, prefix, alignment and padding. These
706 allocators can also be configured in a bufferpool when this is
707 supported by the bufferpool.
712 An array of supported <classname>GstMeta</classname> implementations
713 along with metadata specific parameters.
714 It is important that the upstream element knows what kind of
715 metadata is supported downstream before it places that metadata
721 When the GST_QUERY_ALLOCATION returns, the source pad will select
722 from the available bufferpools, allocators and metadata how it will
726 <sect2 id="section-allocation-query-ex" xreflabel="Allocation-ex">
727 <title>ALLOCATION query example</title>
729 Below is an example of the ALLOCATION query.
733 #include <gst/video/video.h>
734 #include <gst/video/gstvideometa.h>
735 #include <gst/video/gstvideopool.h>
739 GstStructure *structure;
741 GstStructure *config;
742 guint size, min, max;
746 /* find a pool for the negotiated caps now */
747 query = gst_query_new_allocation (caps, TRUE);
749 if (!gst_pad_peer_query (scope->srcpad, query)) {
750 /* query failed, not a problem, we use the query defaults */
753 if (gst_query_get_n_allocation_pools (query) > 0) {
754 /* we got configuration from our peer, parse them */
755 gst_query_parse_nth_allocation_pool (query, 0, &pool, &size, &min, &max);
763 /* we did not get a pool, make one ourselves then */
764 pool = gst_video_buffer_pool_new ();
767 config = gst_buffer_pool_get_config (pool);
768 gst_buffer_pool_config_add_option (config, GST_BUFFER_POOL_OPTION_VIDEO_META);
769 gst_buffer_pool_config_set_params (config, caps, size, min, max);
770 gst_buffer_pool_set_config (pool, config);
773 gst_buffer_pool_set_active (pool, TRUE);
779 This particular implementation will make a custom
780 <classname>GstVideoBufferPool</classname> object that is specialized
781 in allocating video buffers. You can also enable the pool to
782 put <classname>GstVideoMeta</classname> metadata on the buffers from
784 <function>gst_buffer_pool_config_add_option (config,
785 GST_BUFFER_POOL_OPTION_VIDEO_META)</function>.
789 <sect2 id="section-allocation-query-base" xreflabel="Allocation-base">
790 <title>The ALLOCATION query in base classes</title>
792 In many baseclasses you will see the following virtual methods for
793 influencing the allocation strategy:
798 <function>propose_allocation ()</function> should suggest
799 allocation parameters for the upstream element.
804 <function>decide_allocation ()</function> should decide the
805 allocation parameters from the suggestions received from
811 Implementors of these methods should modify the given
812 <classname>GstQuery</classname> object by updating the pool options
813 and allocation options.