Merge remote-tracking branch 'origin/0.10'

[platform/upstream/gstreamer.git] / docs / design / part-bufferpool.txt

Bufferpool
----------

This document details the design of how buffers are be allocated and
managed in pools.

Bufferpools increases performance by reducing allocation overhead and
improving possibilities to implement zero-copy memory transfer.

Together with the ALLOCATION query, elements can negotiate allocation properties
and bufferpools between themselves. This also allows elements to negotiate
buffer metadata between themselves.

Requirements
------------

 - Provide a GstBufferPool base class to help the efficient implementation of a
   list of reusable GstBuffer objects.

 - Let upstream elements initiate the negotiation of a bufferpool and it
   configuration. Allow downstream elements provide bufferpool properties and/or
   a bufferpool. This includes the following properties:

    * have minimum and maximum amount of buffers with the option of 
      preallocating buffers.
    * alignment and padding support
    * buffer metadata
    * arbitrary extra options

 - Integrate with dynamic caps renegotiation.

 - Notify upstream element of new bufferpool availability. This is important
   when a new element, that can provide a bufferpool, is dynamically linked
   downstream.


GstBufferPool
-------------

 The bufferpool object manages a list of buffers with the same properties such
 as size, padding and alignment.

 The bufferpool has two states: active and inactive. In the in-active
 state, the bufferpool can be configured with the required allocation
 preferences. In the active state, buffers can be retrieved from and
 returned to the pool.

 The default implementation of the bufferpool is able to allocate buffers
 from main memory with arbitrary alignment and padding/prefix.

 Custom implementations of the bufferpool can override the allocation and
 free algorithms of the buffers from the pool. This should allow for
 different allocation strategies such as using shared memory or hardware
 mapped memory.


Negotiation
-----------

 After a particular media format has been negotiated between two pads (using the
 CAPS event), they must agree on how to allocate buffers.

 The srcpad will always take the initiative to negotiate the allocation
 properties. It starts with creating a GST_QUERY_ALLOCATION with the negotiated
 caps.

 The srcpad can set the need-pool flag to TRUE in the query to optionally make the
 peer pad allocate a bufferpool.

 It will then inspect the returned results and configure the returned pool or
 create a new pool with the returned properties when needed.

 Buffers are then allocated by the srcpad from the negotiated pool and pushed to
 the peer pad as usual.

 The allocation query can also return an allocator name when the buffers are of
 different sizes and can't be allocated from a pool.


Allocation query
----------------

 The allocation query has the following fields:

  (in) "caps", GST_TYPE_CAPS
        - the caps that was negotiated

  (in) "need-pool", G_TYPE_BOOLEAN
        - if a GstBufferPool is requested

  (out) "size", G_TYPE_UINT
        - the total size of the buffer memory. This size is set to 0 when the 
          buffers are of variable size.

  (out) "min-buffers", G_TYPE_UINT
        - the minimum amount of buffers to allocate

  (out) "max-buffers", G_TYPE_UINT
        - the maximum amount of buffers to allocate

  (out) "prefix", G_TYPE_UINT
        - the prefix of the buffer memory

  (out) "align", G_TYPE_UINT
        - the aligment of the memory in the buffers, the alignment will be applied
          to the prefix. The aligment is expressed as a value ((1 << N) - 1).
          Alignment will be done on multiples of (1 << N).

  (out) "pool", GST_TYPE_BUFFER_POOL
        - a buffer pool when need-pool was TRUE and the peer can provide a pool.
          This pool is inactive and can be configured when needed.

  (out) "metadata", G_TYPE_ARRAY of GType
        - an array of metadata API types that can be accepted.

  (out) "allocator", G_TYPE_ARRAY of GstAllocator *
        - an array of allocators that can be used.


Allocating from pool
--------------------

 Buffers are allocated from the pool of a pad:

  res = gst_buffer_pool_acquire_buffer (pool, &buffer, &params);

 A GstBuffer that is allocated from the pool will always be writable (have a
 refcount of 1) and it will also have its pool member point to the GstBufferPool
 that created the buffer.

 Buffers are refcounted in the usual way. When the refcount of the buffer
 reaches 0, the buffer is automatically returned to the pool.

 Since all the buffers allocated from the pool keep a reference to the pool,
 when nothing else is holding a refcount to the pool, it will be finalized
 when all the buffers from the pool are unreffed. By setting the pool to
 the inactive state we can drain all buffers from the pool. 

 When the pool is in the inactive state, gst_buffer_pool_acquire_buffer() will
 return GST_FLOW_WRONG_STATE immediately.

 Extra parameters can be given to the gst_buffer_pool_acquire_buffer() method to
 influence the allocation decision. GST_BUFFER_POOL_FLAG_KEY_UNIT and
 GST_BUFFER_POOL_FLAG_DISCONT serve as hints.

 When the bufferpool is configured with a maximum number of buffers, allocation
 will block when all buffers are outstanding until a buffer is returned to the
 pool. This behaviour can be changed by specifying the
 GST_BUFFER_POOL_FLAG_DONTWAIT flag in the parameters. With this flag set,
 allocation will return GST_FLOW_EOS when the pool is empty.


Renegotiation
-------------

Renegotiation of the bufferpool might need to be performed when the
configuration of the pool changes. Changes can be in the buffer size (because
of a caps change), alignment or number of buffers. 

* downstream

  When the upstream element wants to negotiate a new format, it might need
  to renegotiate a new bufferpool configuration with the downstream element.
  This can, for example, happen when the buffer size changes.

  We can not just reconfigure the existing bufferpool because there might
  still be outstanding buffers from the pool in the pipeline. Therefore we
  need to create a new bufferpool for the new configuration while we let the
  old pool drain.

  Implementations can choose to reuse the same bufferpool object and wait for
  the drain to finish before reconfiguring the pool.

  The element that wants to renegotiate a new bufferpool uses exactly the same
  algorithm as when it first started. It will negotiate caps first then use the
  ALLOCATION query to get and configure the new pool.
  
* upstream 

  When a downstream element wants to negotiate a new format, it will send a 
  RECONFIGURE event upstream. This instructs upstream to renegotiate both
  the format and the bufferpool when needed.

  A pipeline reconfiguration is when new elements are added or removed from
  the pipeline or when the topology of the pipeline changes. Pipeline
  reconfiguration also triggers possible renegotiation of the bufferpool and
  caps.

  A RECONFIGURE event tags each pad it travels on as needing reconfiguration.
  The next buffer allocation will then require the renegotiation or
  reconfiguration of a pool.


Shutting down
-------------

 In push mode, a source pad is responsible for setting the pool to the
 inactive state when streaming stops. The inactive state will unblock any pending
 allocations so that the element can shut down.

 In pull mode, the sink element should set the pool to the inactive state when
 shutting down so that the peer _get_range() function can unblock.

 In the inactive state, all the buffers that are returned to the pool will
 automatically be freed by the pool and new allocations will fail.
 

Use cases
---------

1)  videotestsrc ! xvimagesink

 Before videotestsrc can output a buffer, it needs to negotiate caps and
 a bufferpool with the downstream peer pad.

 First it will negotiate a suitable format with downstream according to the
 normal rules. It will send a CAPS event downstream with the negotiated
 configuration.

 Then it does an ALLOCATION query. It will use the returned bufferpool or
 configures its own bufferpool with the returned parameters. The bufferpool is
 initially in the inactive state.

 The ALLOCATION query lists the desired configuration of the downstream 
 xvimagesink, which can have specific alignment and/or min/max amount of
 buffers.

 videotestsrc updates the configuration of the bufferpool, it will likely
 set the min buffers to 1 and the size of the desired buffers. It then
 updates the bufferpool configuration with the new properties.

 When the configuration is successfully updated, videotestsrc sets the
 bufferpool to the active state. This preallocates the buffers in the pool
 (if needed). This operation can fail when there is not enough memory
 available. Since the bufferpool is provided by xvimagesink, it will allocate
 buffers backed by an XvImage and pointing to shared memory with the X server.

 If the bufferpool is successfully activated, videotestsrc can acquire a
 buffer from the pool, fill in the data and push it out to xvimagesink.

 xvimagesink can know that the buffer originated from its pool by following
 the pool member.

 when shutting down, videotestsrc will set the pool to the inactive state,
 this will cause further allocations to fail and currently allocated buffers
 to be freed. videotestsrc will then free the pool and stop streaming.


2) videotestsrc ! queue ! myvideosink

 In this second use case we have a videosink that can at most allocate
 3 video buffers.

 Again videotestsrc will have to negotiate a bufferpool with the peer 
 element. For this it will perform the ALLOCATION query which
 queue will proxy to its downstream peer element.

 The bufferpool returned from myvideosink will have a max_buffers set to 3.
 queue and videotestsrc can operate with this upper limit because none of
 those elements require more than that amount of buffers for temporary
 storage.

 The bufferpool of myvideosink will then be configured with the size of the
 buffers for the negotiated format and according to the padding and alignment
 rules. When videotestsrc sets the pool to active, the 3 video
 buffers will be preallocated in the pool.

 videotestsrc acquires a buffer from the configured pool on its srcpad and
 pushes this into the queue. When the videotestsrc has acquired and pushed
 3 frames, the next call to gst_buffer_pool_acquire_buffer() will block
 (assuming the GST_BUFFER_POOL_FLAG_DONTWAIT is not specified).

 When the queue has pushed out a buffer and the sink has rendered it, the
 refcount of the buffer reaches 0 and the buffer is recycled in the pool.
 This will wake up the videotestsrc that was blocked, waiting for more
 buffers and will make it produce the next buffer. 

 In this setup, there are at most 3 buffers active in the pipeline and
 the videotestsrc is rate limited by the rate at which buffers are recycled
 in the bufferpool.

 When shutting down, videotestsrc will first set the bufferpool on the srcpad
 to inactive. This causes any pending (blocked) acquire to return with a 
 WRONG_STATE result and causes the streaming thread to pause.

 
3) .. ! myvideodecoder ! queue ! fakesink 

 In this case, the myvideodecoder requires buffers to be aligned to 128
 bytes and padded with 4096 bytes. The pipeline starts out with the
 decoder linked to a fakesink but we will then dynamically change the
 sink to one that can provide a bufferpool.

 When it negotiates the size with the downstream element fakesink, it will
 receive a NULL bufferpool because fakesink does not provide a bufferpool.
 It will then select its own custom bufferpool to start the datatransfer.

 At some point we block the queue srcpad, unlink the queue from the 
 fakesink, link a new sink and set the new sink to the PLAYING state.
 Linking the new sink would automatically send a RECONFIGURE event upstream
 and, through queue, inform myvideodecoder that it should renegotiate its
 bufferpool because downstream has been reconfigured.

 Before pushing the next buffer, myvideodecoder would renegotiate a new
 bufferpool. To do this, it performs the usual bufferpool negotiation
 algorithm. If it can obtain and configure a new bufferpool from downstream,
 it sets its own (old) pool to inactive and unrefs it. This will eventually
 drain and unref the old bufferpool.

 The new bufferpool is set as the new bufferpool for the srcpad and sinkpad
 of the queue and set to the active state.


4) .. ! myvideodecoder ! queue ! myvideosink 

 myvideodecoder has negotiated a bufferpool with the downstream myvideosink
 to handle buffers of size 320x240. It has now detected a change in the 
 video format and need to renegotiate to a resolution of 640x480. This
 requires it to negotiate a new bufferpool with a larger buffersize.

 When myvideodecoder needs to get the bigger buffer, it starts the 
 negotiation of a new bufferpool. It queries a bufferpool from downstream,
 reconfigures it with the new configuration (which includes the bigger buffer
 size) and it then sets the bufferpool to active. The old pool is inactivated
 and unreffed, which causes the old format to drain.

 It then uses the new bufferpool for allocating new buffers of the new 
 dimension.

 If at some point, the decoder wants to switch to a lower resolution again,
 it can choose to use the current pool (which has buffers that are larger
 than the required size) or it can choose to renegotiate a new bufferpool.


5) .. ! myvideodecoder ! videoscale ! myvideosink 

 myvideosink is providing a bufferpool for upstream elements and wants to
 change the resolution.

 myvideosink sends a RECONFIGURE event upstream to notify upstream that a
 new format is desirable. upstream elements try to negotiate a new format
 and bufferpool before pushing out a new buffer. The old bufferpools are
 drained in the regular way.