1 <chapter id="chapter-allocation" xreflabel="Memory allocation">
2 <title>Memory allocation</title>
4 Memory allocation and management is a very important topic in
5 multimedia. High definition video uses many magabytes to store
6 one single frame of video. It is important to reuse the memory
7 when possible instead of constantly allocating and freeing
11 Multimedia systems usually use special purpose chips, such as
12 DSPs or GPUs to perform the heavy lifting (especially for video).
13 These special purpose chips have usually strict requirements
14 for the memory that they can operate on and how the memory
18 This chapter talks about the memory management features that
19 &GStreamer; plugins can use. We will first talk about the
20 lowlevel <classname>GstMemory</classname> object that manages
21 access to a piece of memory. We then continue with
22 <classname>GstBuffer</classname> that is used to exchange data
23 between plugins (and the application) and that uses
24 <classname>GstMemory</classname>. We talk about
25 <classname>GstMeta</classname> that can be placed on buffers to
26 give extra info about the buffer and its memory.
27 For efficiently managing buffers of the same size, we take a
28 look at <classname>GstBufferPool</classname>. To conclude this
29 chapter we take a look at the GST_QUERY_ALLOCATION query that
30 is used to negotiate memory management options between elements.
33 <sect1 id="section-allocation-memory" xreflabel="GstMemory">
34 <title>GstMemory</title>
36 <classname>GstMemory</classname> is an object that manages a region
37 of memory. The memory object points to a region of memory of
38 <quote>maxsize</quote>. The area in this memory starting at
39 <quote>offset</quote> and for <quote>size</quote> bytes is the
40 accessible region in the memory. the maxsize of the memory can
41 never be changed after the object is created, however, the offset
42 and size can be changed.
45 <classname>GstMemory</classname> objects are created by a
46 <classname>GstAllocator</classname> object. To implement support
47 for a new kind of memory type, you must implement a new allocator
50 <sect2 id="section-allocation-memory-ex" xreflabel="GstMemory-ex">
51 <title>GstMemory API example</title>
53 Data access to the memory wrapped by the <classname>GstMemory</classname>
54 object is always protected with a <function>gst_memory_map()</function>
55 and <function>gst_memory_unmap()</function> pair. An access mode
56 (read/write) must be given when mapping memory. The map
57 function returns a pointer to the valid memory region that can
58 then be accessed according to the requested access mode.
61 Below is an example of making a <classname>GstMemory</classname>
62 object and using the <function>gst_memory_map()</function> to
63 access the memory region.
73 /* allocate 100 bytes */
74 mem = gst_allocator_alloc (NULL, 100, NULL);
76 /* get access to the memory in write mode */
77 gst_memory_map (mem, &info, GST_MAP_WRITE);
79 /* fill with pattern */
80 for (i = 0; i < info.size; i++)
84 gst_memory_unmap (mem, &info);
91 <sect2 id="section-allocation-allocator" xreflabel="GstAllocator">
92 <title>Implementing a GstAllocator</title>
100 <sect1 id="section-allocation-buffer" xreflabel="GstBuffer">
101 <title>GstBuffer</title>
103 A <classname>GstBuffer</classname> is an lightweight object that
104 is passed from an upstream to a downstream element and contains
105 memory and metadata. It represents the multimedia content that
106 is pushed or pull downstream by elements.
109 The buffer contains one or more <classname>GstMemory</classname>
110 objects thet represent the data in the buffer.
113 Metadata in the buffer consists of:
115 <itemizedlist mark="opencircle">
118 DTS and PTS timestamps. These represent the decoding and
119 presentation timestamps of the buffer content and is used by
120 synchronizing elements to schedule buffers. Both these timestamps
121 can be GST_CLOCK_TIME_NONE when unknown/undefined.
126 The duration of the buffer contents. This duration can be
127 GST_CLOCK_TIME_NONE when unknown/undefined.
132 Media specific offsets and offset_end. For video this is the
133 frame number in the stream and for audio the sample number. Other
134 definitions for other media exist.
139 Arbitrary structures via <classname>GstMeta</classname>, see below.
144 <sect2 id="section-allocation-writability" xreflabel="GstBuffer-write">
145 <title>GstBuffer writability</title>
147 A buffer is writable when the refcount of the object is exactly 1, meaning
148 that only one object is holding a ref to the buffer. You can only
149 modify anything in the buffer when the buffer is writable. This means
150 that you need to call <function>gst_buffer_make_writable()</function>
151 before changing the timestamps, offsets, metadata or adding and
152 removing memory blocks.
155 <sect2 id="section-allocation-buffer-ex" xreflabel="GstBuffer-ex">
156 <title>GstBuffer API examples</title>
158 You can create a buffer with <function>gst_buffer_new ()</function>
159 and then add memory objects to it or you can use a convenience function
160 <function>gst_buffer_new_allocate ()</function> which combines the
161 two. It's also possible to wrap existing memory with
162 <function>gst_buffer_new_wrapped_full () </function> where you can
163 give the function to call when the memory should be freed.
166 You can access the memory of the buffer by getting and mapping the
167 <classname>GstMemory</classname> objects individually or by using
168 <function>gst_buffer_map ()</function>. The latter merges all the
169 memory into one big block and then gives you a pointer to this block.
172 Below is an example of how to create a buffer and access its memory.
181 /* make empty buffer */
182 buffer = gst_buffer_new ();
184 /* make memory holding 100 bytes */
185 mem = gst_allocator_alloc (NULL, 100, NULL);
187 /* add the the buffer */
188 gst_buffer_append_memory (buffer, mem);
192 /* get WRITE access to the memory and fill with 0xff */
193 gst_buffer_map (buffer, &info, GST_MAP_WRITE);
194 memset (info.data, 0xff, info.size);
195 gst_buffer_unmap (buffer, &info);
199 /* free the buffer */
200 gst_buffer_unref (buffer);
208 <sect1 id="section-allocation-meta" xreflabel="GstMeta">
209 <title>GstMeta</title>
211 With the <classname>GstMeta</classname> system you can add arbitrary
212 structures of on buffers. These structures describe extra properties
213 of the buffer such as cropping, stride, region of interest etc.
216 Metadata is also used to store, for example, the X image that is
217 backing up the memory of the buffer. This makes it easier for elements
218 to locate the X image from the buffer.
221 The metadata system separates API specification (what the metadata
222 and its API look like) and the implementation (how it works). This makes
223 it possible to make different implementations of the same API,
224 for example, depending on the hardware you are running on.
227 <sect2 id="section-allocation-meta-ex" xreflabel="GstMeta-ex">
228 <title>GstMeta API example</title>
230 After allocating a new buffer, you can add metadata to the buffer
231 with the metadata specific API. This means that you will need to
232 link to the header file where the metadata is defined to use
236 By convention, a metadata API with name <classname>FooBar</classname>
237 should provide two methods, a
238 <function>gst_buffer_add_foo_bar_meta ()</function> and a
239 <function>gst_buffer_get_foo_bar_meta ()</function>. Both functions
240 should return a pointer to a <classname>FooBarMeta</classname>
241 structure that contains the metadata fields. Some of the
242 <function>_add_*_meta ()</function> can have extra parameters that
243 will usually be used to configure the metadata structure for you.
246 Let's have a look at the metadata that is used to specify a cropping
247 region for video frames.
251 #include <gst/video/gstvideometa.h>
254 GstVideoCropMeta *meta;
256 /* buffer points to a video frame, add some cropping metadata */
257 meta = gst_buffer_add_video_crop_meta (buffer);
259 /* configure the cropping metadata */
268 An element can then use the metadata on the buffer when rendering
273 #include <gst/video/gstvideometa.h>
276 GstVideoCropMeta *meta;
278 /* buffer points to a video frame, get the cropping metadata */
279 meta = gst_buffer_get_video_crop_meta (buffer);
282 /* render frame with cropping */
283 _render_frame_cropped (buffer, meta->x, meta->y, meta->width, meta->height);
286 _render_frame (buffer);
294 <sect2 id="section-allocation-meta-new" xreflabel="GstMeta-new">
295 <title>Implementing new GstMeta</title>
297 In the next sections we show how you can add new metadata to the
298 system and use it on buffers.
301 <sect3 id="section-allocation-meta-api" xreflabel="GstMeta-api">
302 <title>Define the metadata API</title>
304 First we need to define what our API will look like and we
305 will have to register this API to the system. This is important
306 because this API definition will be used when elements negotiate
307 what kind of metadata they will exchange. The API definition
308 also contains arbitrary tags that give hints about what the
309 metadata contains. This is important when we see how metadata
310 is preserved when buffers pass through the pipeline.
313 If you are making a new implementation of an existing API,
314 you can skip this step and move on to the implementation step.
317 First we start with making the
318 <filename>my-example-meta.h</filename> header file that will contain
319 the definition of the API and structure for our metadata.
325 typedef struct _MyExampleMeta MyExampleMeta;
327 struct _MyExampleMeta {
334 GType my_example_meta_api_get_type (void);
335 #define MY_EXAMPLE_META_API_TYPE (my_example_meta_api_get_type())
337 #define gst_buffer_get_my_example_meta(b) \
338 ((MyExampleMeta*)gst_buffer_get_meta((b),MY_EXAMPLE_META_API_TYPE))
342 The metadata API definition consists of the definition of the
343 structure that holds a gint and a string. The first field in
344 the structure must be <classname>GstMeta</classname>.
347 We also define a <function>my_example_meta_api_get_type ()</function>
348 function that will register out metadata API definition. We
349 also define a convenience macro
350 <function>gst_buffer_get_my_example_meta ()</function> that simply
351 finds and returns the metadata with our new API.
354 Next let's have a look at how the
355 <function>my_example_meta_api_get_type ()</function> function is
356 implemented in the <filename>my-example-meta.c</filename> file.
360 #include "my-example-meta.h"
363 my_example_meta_api_get_type (void)
365 static volatile GType type;
366 static const gchar *tags[] = { "foo", "bar", NULL };
368 if (g_once_init_enter (&type)) {
369 GType _type = gst_meta_api_type_register ("MyExampleMetaAPI", tags);
370 g_once_init_leave (&type, _type);
377 As you can see, it simply uses the
378 <function>gst_meta_api_type_register ()</function> function to
379 register a name for the api and some tags. The result is a
380 new pointer GType that defines the newly registered API.
384 <sect3 id="section-allocation-meta-impl" xreflabel="GstMeta-impl">
385 <title>Implementing a metadata API</title>
387 Next we can make an implementation for a registered metadata
388 API GType. The implementation detail of a metadata API
389 are kept in a <classname>GstMetaInfo</classname> structure
390 that you will make available to the users of your metadata
391 API implementation with a <function>my_example_meta_get_info ()</function>
392 function and a convenience <function>MY_EXAMPLE_META_INFO</function>
393 macro. You will also make a method to add your metadata
394 implementation to a <classname>GstBuffer</classname>.
395 Your <filename>my-example-meta.h</filename> header file will
403 const GstMetaInfo *my_example_meta_get_info (void);
404 #define MY_EXAMPLE_META_INFO (my_example_meta_get_info())
406 MyExampleMeta * gst_buffer_add_my_example_meta (GstBuffer *buffer,
412 Let's have a look at how these functions are
413 implemented in the <filename>my-example-meta.c</filename> file.
420 my_example_meta_init (GstMeta * meta, gpointer params, GstBuffer * buffer)
422 MyExampleMeta *emeta = (MyExampleMeta *) meta;
431 my_example_meta_transform (GstBuffer * transbuf, GstMeta * meta,
432 GstBuffer * buffer, GQuark type, gpointer data)
434 MyExampleMeta *emeta = (MyExampleMeta *) meta;
436 /* we always copy no matter what transform */
437 gst_buffer_add_my_example_meta (transbuf, emeta->age, emeta->name);
443 my_example_meta_free (GstMeta * meta, GstBuffer * buffer)
445 MyExampleMeta *emeta = (MyExampleMeta *) meta;
452 my_example_meta_get_info (void)
454 static const GstMetaInfo *meta_info = NULL;
456 if (g_once_init_enter (&meta_info)) {
457 const GstMetaInfo *mi = gst_meta_register (MY_EXAMPLE_META_API_TYPE,
459 sizeof (MyExampleMeta),
460 my_example_meta_init,
461 my_example_meta_free,
462 my_example_meta_transform);
463 g_once_init_leave (&meta_info, mi);
469 gst_buffer_add_my_example_meta (GstBuffer *buffer,
475 g_return_val_if_fail (GST_IS_BUFFER (buffer), NULL);
477 meta = (MyExampleMeta *) gst_buffer_add_meta (buffer,
478 MY_EXAMPLE_META_INFO, NULL);
481 meta->name = g_strdup (name);
488 <function>gst_meta_register ()</function> registers the implementation
489 details, like the API that you implement and the size of the
490 metadata structure along with methods to initialize and free the
491 memory area. You can also implement a transform function that will
492 be called when a certain transformation (identified by the quark and
493 quark specific data) is performed on a buffer.
496 Lastly, you implement a <function>gst_buffer_add_*_meta()</function>
497 that adds the metadata implementation to a buffer and sets the
498 values of the metadata.
505 <sect1 id="section-allocation-bufferpool" xreflabel="GstBufferPool">
506 <title>GstBufferPool</title>
508 The <classname>GstBufferPool</classname> object provides a convenient
509 base class for managing lists of reusable buffers. Essential for this
510 object is that all the buffers have the same properties such as size,
511 padding, metadata and alignment.
514 A bufferpool object can be configured to manage a minimum and maximum
515 amount of buffers of a specific size. A bufferpool can also be
516 configured to use a specific <classname>GstAllocator</classname> for
517 the memory of the buffers. There is support in the bufferpool to enable
518 bufferpool specific options, such as adding <classname>GstMeta</classname>
519 to the buffers in the pool or such as enabling specific padding on
520 the memory in the buffers.
523 A Bufferpool can be inactivate and active. In the inactive state,
524 you can configure the pool. In the active state, you can't change
525 the configuration anymore but you can acquire and release buffers
529 In the following sections we take a look at how you can use
533 <sect2 id="section-allocation-pool-ex" xreflabel="GstBufferPool-ex">
534 <title>GstBufferPool API example</title>
536 Many different bufferpool implementations can exist; they are all
537 subclasses of the base class <classname>GstBufferPool</classname>.
538 For this example, we will assume we somehow have access to a
539 bufferpool, either because we created it ourselves or because
540 we were given one as a result of the ALLOCATION query as we will
544 The bufferpool is initially in the inactive state so that we can
545 configure it. Trying to configure a bufferpool that is not in the
546 inactive state will fail. Likewise, trying to activate a bufferpool
547 that is not configured will fail.
551 GstStructure *config;
555 /* get config structure */
556 config = gst_buffer_pool_get_config (pool);
558 /* set caps, size, minimum and maximum buffers in the pool */
559 gst_buffer_pool_config_set_params (config, caps, size, min, max);
561 /* configure allocator and parameters */
562 gst_buffer_pool_config_set_allocator (config, allocator, ¶ms);
564 /* store the updated configuration again */
565 gst_buffer_pool_set_config (pool, config);
571 The configuration of the bufferpool is maintained in a generic
572 <classname>GstStructure</classname> that can be obtained with
573 <function>gst_buffer_pool_get_config()</function>. Convenience
574 methods exist to get and set the configuration options in this
575 structure. After updating the structure, it is set as the current
576 configuration in the bufferpool again with
577 <function>gst_buffer_pool_set_config()</function>.
580 The following options can be configured on a bufferpool:
582 <itemizedlist mark="opencircle">
585 The caps of the buffers to allocate.
590 The size of the buffers. This is the suggested size of the
591 buffers in the pool. The pool might decide to allocate larger
592 buffers to add padding.
597 The minimum and maximum amount of buffers in the pool. When
598 minimum is set to > 0, the bufferpool will pre-allocate this
599 amount of buffers. When maximum is not 0, the bufferpool
600 will allocate up to maximum amount of buffers.
605 The allocator and parameters to use. Some bufferpools might
606 ignore the allocator and use its internal one.
611 Other arbitrary bufferpool options identified with a string.
612 a bufferpool lists the supported options with
613 <function>gst_buffer_pool_get_options()</function> and you
614 can ask if an option is supported with
615 <function>gst_buffer_pool_has_option()</function>. The option
616 can be enabled by adding it to the configuration structure
617 with <function>gst_buffer_pool_config_add_option ()</function>.
618 These options are used to enable things like letting the
619 pool set metadata on the buffers or to add extra configuration
620 options for padding, for example.
625 After the configuration is set on the bufferpool, the pool can
627 <function>gst_buffer_pool_set_active (pool, TRUE)</function>. From
628 that point on you can use
629 <function>gst_buffer_pool_acquire_buffer ()</function> to retrieve
630 a buffer from the pool, like this:
639 ret = gst_buffer_pool_acquire_buffer (pool, &buffer, NULL);
640 if (G_UNLIKELY (ret != GST_FLOW_OK))
647 It is important to check the return value of the acquire function
648 because it is possible that it fails: When your
649 element shuts down, it will deactivate the bufferpool and then
650 all calls to acquire will return GST_FLOW_FLUSHNG.
653 All buffers that are acquired from the pool will have their pool
654 member set to the original pool. When the last ref is decremented
655 on the buffer, &GStreamer; will automatically call
656 <function>gst_buffer_pool_release_buffer()</function> to release
657 the buffer back to the pool. You (or any other downstream element)
658 don't need to know if a buffer came from a pool, you can just
663 <sect2 id="section-allocation-pool-impl" xreflabel="GstBufferPool-impl">
664 <title>Implementing a new GstBufferPool</title>
672 <sect1 id="section-allocation-query" xreflabel="GST_QUERY_ALLOCATION">
673 <title>GST_QUERY_ALLOCATION</title>
675 The ALLOCATION query is used to negotiate
676 <classname>GstMeta</classname>, <classname>GstBufferPool</classname>
677 and <classname>GstAllocator</classname> between elements. Negotiation
678 of the allocation strategy is always initiated and decided by a srcpad
679 after it has negotiated a format and before it decides to push buffers.
680 A sinkpad can suggest an allocation strategy but it is ultimately the
681 source pad that will decide based on the suggestions of the downstream
685 The source pad will do a GST_QUERY_ALLOCATION with the negotiated caps
686 as a parameter. This is needed so that the downstream element knows
687 what media type is being handled. A downstream sink pad can answer the
688 allocation query with the following results:
690 <itemizedlist mark="opencircle">
693 An array of possible <classname>GstBufferPool</classname> suggestions
694 with suggested size, minimum and maximum amount of buffers.
699 An array of GstAllocator objects along with suggested allocation
700 parameters such as flags, prefix, alignment and padding. These
701 allocators can also be configured in a bufferpool when this is
702 supported by the bufferpool.
707 An array of supported <classname>GstMeta</classname> implementations
708 along with metadata specific parameters.
709 It is important that the upstream element knows what kind of
710 metadata is supported downstream before it places that metadata
716 When the GST_QUERY_ALLOCATION returns, the source pad will select
717 from the available bufferpools, allocators and metadata how it will
721 <sect2 id="section-allocation-query-ex" xreflabel="Allocation-ex">
722 <title>ALLOCATION query example</title>
724 Below is an example of the ALLOCATION query.
728 #include <gst/video/video.h>
729 #include <gst/video/gstvideometa.h>
730 #include <gst/video/gstvideopool.h>
734 GstStructure *structure;
736 GstStructure *config;
737 guint size, min, max;
741 /* find a pool for the negotiated caps now */
742 query = gst_query_new_allocation (caps, TRUE);
744 if (!gst_pad_peer_query (scope->srcpad, query)) {
745 /* query failed, not a problem, we use the query defaults */
748 if (gst_query_get_n_allocation_pools (query) > 0) {
749 /* we got configuration from our peer, parse them */
750 gst_query_parse_nth_allocation_pool (query, 0, &pool, &size, &min, &max);
758 /* we did not get a pool, make one ourselves then */
759 pool = gst_video_buffer_pool_new ();
762 config = gst_buffer_pool_get_config (pool);
763 gst_buffer_pool_config_add_option (config, GST_BUFFER_POOL_OPTION_VIDEO_META);
764 gst_buffer_pool_config_set_params (config, caps, size, min, max);
765 gst_buffer_pool_set_config (pool, config);
768 gst_buffer_pool_set_active (pool, TRUE);
774 This particular implementation will make a custom
775 <classname>GstVideoBufferPool</classname> object that is specialized
776 in allocating video buffers. You can also enable the pool to
777 put <classname>GstVideoMeta</classname> metadata on the buffers from
779 <function>gst_buffer_pool_config_add_option (config,
780 GST_BUFFER_POOL_OPTION_VIDEO_META)</function>.
784 <sect2 id="section-allocation-query-base" xreflabel="Allocation-base">
785 <title>The ALLOCATION query in base classes</title>
787 In many baseclasses you will see the following virtual methods for
788 influencing the allocation strategy:
793 <function>propose_allocation ()</function> should suggest
794 allocation parameters for the upstream element.
799 <function>decide_allocation ()</function> should decide the
800 allocation parameters from the suggestions received from
806 Implementors of these methods should modify the given
807 <classname>GstQuery</classname> object by updating the pool options
808 and allocation options.