4 Transform elements transform input buffers to output buffers based
5 on the sink and source caps.
7 An important requirement for a transform is that the ouput caps are completely
8 defined by the input caps and vice versa. This means that a typical decoder
9 element can NOT be implemented with a transform element, this is because the
10 output caps like width and height of the decompessed video frame, for example,
11 are endcoded in the stream and thus not defined by the input caps.
13 Typical transform elements include:
15 - audio convertors (audioconvert, audioresample,...)
16 - video convertors (colorspace, videoscale, ...)
17 - filters (capfilter, volume, colorbalance, ...)
19 The implementation of the transform element has to take care of
22 - efficient negotiation both up and downstream
23 - efficient buffer alloc and other buffer management
25 Some transform elements can operate in different modes:
27 - passthrough (no changes are done on the input buffers)
28 - in-place (changes made directly to the incomming buffers without requiring a
29 copy or new buffer allocation)
30 - metadata changes only
32 Depending on the mode of operation the buffer allocation strategy might change.
34 The transform element should at any point be able to renegotiate sink and src
35 caps as well as change the operation mode.
37 In addition, the transform element will typically take care of the following
42 - timestamping, this is typically done by copying the input timestamps to the
43 output buffers but subclasses should be able to override this.
44 - QoS, avoiding calls to the subclass transform function
45 - handle scheduling issues such as push and pull based operation.
47 In the next sections, we will describe the behaviour of the transform element in
48 each of the above use cases. We focus mostly on the buffer allocation strategies
54 A transform has 2 main processing functions:
58 Transform the input buffer to the output buffer. The output buffer is
59 guaranteed to be writable and different from the input buffer.
63 Transform the input buffer in-place. The input buffer is writable and of
64 bigger or equal size than the output buffer.
66 A transform can operate in the following modes:
70 The element will not make changes to the buffers, buffers are pushed straight
71 through, caps on both sides need to be the same. The element can optionally
72 implement a transform_ip() function to take a look at the data, the buffer
73 does not have to be writable.
77 Changes can be made to the input buffer directly to obtain the output buffer.
78 The transform must implement a transform_ip() function.
82 The transform is performed by copying and transforming the input buffer to a
83 new output buffer. The transform must implement a transform() function.
85 When no transform() function is provided, only in-place and passthrough
86 operation is allowed, this means that source and destination caps must be equal
87 or that the source buffer size is bigger or equal than the destination buffer.
89 When no transform_ip() function is provided, only passthrough and
90 copy-transforms are supported. Providing this function is an optimisation that
91 can avoid a buffer copy.
93 When no functions are provided, we can only process in passthrough mode.
99 Typical (re)negotiation of the transform element in push mode always goes from
100 sink to src, this means triggers the following sequence:
102 - the sinkpad receives a buffer with new caps, this triggers the setcaps
103 function on the sinkpad before handing the buffer to transform.
104 - the transform function figures out what it can convert these caps to.
105 - try to see if we can configure the caps unmodified on the peer. We need to
106 do this because we prefer to not do anything.
107 - the transform configures itself to transform from the new sink caps to the
109 - the transform processes and sets the output caps on the src pad
111 We call this downstream negotiation (DN) and it goes roughly like this:
113 sinkpad transform srcpad
115 ------------>| find_transform() | |
116 |------------------->| |
118 | |--------------------->|
119 | <configure caps> <-| |
122 These steps configure the element for a transformation from the input caps to
125 The transform has 3 function to perform the negotiation:
129 Transform the caps on a certain pad to all the possible supported caps on
130 the other pad. The input caps are guaranteed to be a simple caps with just
131 one structure. The caps do not have to be fixed.
135 Given a caps on one pad, fixate the caps on the other pad. The target caps
140 Configure the transform for a transformation between src caps and dest
141 caps. Both caps are guaranteed to be fixed caps.
143 If no transform_caps() is defined, we can only perform the identity transform,
146 If no set_caps() is defined, we don't care about caps. In that case we also
147 assume nothing is going to write to the buffer and we don't enforce a writable
148 buffer for the transform_ip function, when present.
150 One common function that we need for the transform element is to find the best
151 transform from one format (src) to another (dest). Since the function is
152 bidirectional, we will use the src->dest negotiation. Some requirements of this
155 - has a fixed src caps
156 - finds a fixed dest caps that the transform element can transform to
157 - the dest caps are compatible and can be accepted by peer elements
158 - the transform function prefers to make src caps == dest caps
159 - the transform function can optionally fixate dest caps.
161 The find_transform() function goes like this:
163 - start from src aps, these caps are fixed.
164 - check if the caps are acceptable for us as src caps. This is usually
165 enforced by the padtemplate of the element.
166 - calculate all caps we can transform too with transform_caps()
167 - if the original caps are a subset of the transforms, try to see if the
168 the caps are acceptable for the peer. If this is possible, we can
169 perform passthrough and make src == dest. This is performed by simply
170 calling gst_pad_peer_accept_caps().
171 - if the caps are not fixed, we need to fixate it, start by taking the peer
172 caps and intersect with them.
173 - for each of the transformed caps retrieved with transform_caps():
174 - try to fixate the caps with fixate_caps()
175 - if the caps are fixated, check if the peer accepts them with
176 _peer_accept_caps(), if the peer accepts, we have found a dest caps.
177 - if we run out of caps, we fail to find a transform.
178 - if we found a destination caps, configure the transform with set_caps().
180 After this negotiation process, the transform element is usually in a steady
181 state. We can identify these steady states:
183 - src and sink pads both have the same caps. Note that when the caps are equal
184 on both pads, the input and output buffers automatically have the same size.
185 The element can operate on the buffers in the following ways: (Same caps, SC)
187 - passthrough: buffers are inspected but no metadata or buffer data
188 is changed. The input buffers don't need to be writable. The input
189 buffer is simply pushed out again without modifications. (SCP)
191 sinkpad transform srcpad
193 ------------>| handle_buffer() | |
194 |------------------->| pad_push() |
195 | |--------------------->|
198 - in-place: buffers are modified in-place, this means that the input
199 buffer is modified to produce a new output buffer. This requires the
200 input buffer to be writable. If the input buffer is not writable, a new
201 buffer has to be allocated with pad-alloc. (SCI)
203 sinkpad transform srcpad
205 ------------>| handle_buffer() | |
206 |------------------->| |
209 | |--------------------->|
210 | [caps-changed] .-| [caps-changed] |
211 | <reconfigure> | | setcaps() |
212 | '>|--------------------->|
214 | <transform_ip> | | |
217 | |--------------------->|
220 - copy transform: a new output buffer is allocated with pad-alloc and data
221 from the input buffer is transformed into the output buffer. (SCC)
223 sinkpad transform srcpad
225 ------------>| handle_buffer() | |
226 |------------------->| |
228 | |--------------------->|
229 | [caps-changed] .-| [caps-changed] |
230 | <reconfigure> | | setcaps() |
231 | '>|--------------------->|
236 | |--------------------->|
239 - src and sink pads have different caps. The element can operate on the
240 buffers in the following way: (Different Caps, DC)
242 - in-place: input buffers are modified in-place. This means that the input
243 buffer has a size that is larger or equal to the output size. The input
244 buffer will be resized to the size of the output buffer. If the input
245 buffer is not writable or the output size is bigger than the input size,
246 we need to pad-alloc a new buffer. (DCI)
248 sinkpad transform srcpad
250 ------------>| handle_buffer() | |
251 |------------------->| |
252 | | [!writable || !size] |
254 | |--------------------->|
255 | [caps-changed] .-| [caps-changed] |
256 | <reconfigure> | | setcaps() |
257 | '>|--------------------->|
259 | <transform_ip> | | |
262 | |--------------------->|
265 - copy transform: a new output buffer is allocated and the data from the
266 input buffer is transformed into the output buffer. The flow is exactly
267 the same as the case with the same-caps negotiation. (DCC)
269 We can immeditatly observe that the copy transform states will need to
270 allocate a buffer from a downstream element using pad-alloc. When the transform
271 element is receiving a non-writable buffer in the in-place state, it will also
272 need to perform a pad-alloc. There is no reason why the passthrough state would
273 perform a pad-alloc. This is important because upstream re-negotiation can only
274 happen when the transform uses pad-alloc for all outgoing buffers.
276 This steady state changes when one of the following actions occur:
278 - the sink pad receives new caps, this triggers the above downstream
279 renegotation process, see above for the flow.
280 - the src pad is instructed to produce new caps because of new caps from
281 pad-alloc, this only happens when the transform calls pad-alloc on the
282 srcpad in order to produce a new output buffer.
283 - the transform element wants to renegotiate (because of changed properties,
284 for example). This essentially clears the current steady state and
285 triggers the downstream and upstream renegotiation process.
287 Parallel to the downstream negotiation process there is an upstream negotiation
288 process. The handling and proxy of buffer-alloc is the most comple part of the
289 transform element. This upstream negotiation process has 3 cases: (UN)
291 - upstream calls the buffer-alloc function of the transform sinkpad and this
292 call is proxied downstream (UNP)
293 - upstream calls the buffer-alloc function of the transform sinkpad, the
294 transform does not proxy the call but returns a buffer itself (UNU)
295 - the transform calls the pad-alloc function downstream to allocate a new
296 output buffer (but not because of a proxied buffer-alloc) (UNA)
298 The case where the pad-alloc is called because an output buffer must be
299 generated in the chain function is handled above in the copy-transform and the
300 in-place transform when the input buffer is not writable or the input buffer
301 size is smaller than the output size.
303 We are left with the last case (proxy an incomming pad-alloc or not). We have 2
306 - pad-alloc is called with the same caps as are currently being handled by
307 the transform on the sinkcaps. Note that this will only be true when the
308 transform element is completely negotiated because of data processing, see
309 above. Then the element is not yet negotiated, we proceed with the case
310 where sinkcaps are different from thos in the buffer-alloc.
312 * If the transform is using copy-transform, we don't need to proxy because
313 we will call pad-alloc when generating an output buffer.
315 sinkpad transform srcpad
317 --------------->| | |
319 |-. [same caps && | |
320 return default | | copy-trans] | |
324 * If the transform is using in-place and insize < outsize, we proxy
325 the pad-alloc with the srccaps. If the caps are unmodified, we proxy
326 the buffer after changing the caps and size.
328 sinkpad transform srcpad
330 --------------->| | |
333 |------------------->| pad_alloc() |
334 | |--------------------->|
335 | [caps unchanged] | |
336 return | adjust_buffer | |
337 <----------------------------------| |
341 * If the transform is using in-place and insize < outsize, we proxy
342 the pad-alloc with the srccaps. If the caps are modified find the best
343 transform from these new caps and return a buffer of this size/caps
346 sinkpad transform srcpad
348 --------------->| | |
350 | in-place] | pad-alloc() |
351 |------------------------------------------>|
352 | [caps changed] .-| |
353 | find_transform() | | |
355 <----------------------------------| |
358 * If the transform is using in-place and insize >= outsize, we cannot proxy
359 the pad-alloc because the resulting buffer would be too small to return
362 * If the transform is using passthrough, we can proxy the pad-alloc to the
363 source pad. If the caps change, find the best transform and return a
364 buffer of those caps and size instead.
366 sinkpad transform srcpad
368 --------------->| [same caps && | |
369 | passtrough] | pad-alloc() |
370 |------------------------------------------>|
371 | [caps changed] .-| |
372 | find_transform() | | |
374 <----------------------------------| |
377 - pad-alloc is called with different caps than are currently being handled by
378 the transform on the sinkcaps we have to try to negotiate a new
379 configuration for the transform element.
381 * we perform the standard way to finding a best transform using
382 find_transform() and we call the pad-alloc function with these caps.
383 If we get different caps from pad-alloc, we find the best format to
384 transform these to and return those caps instead.
387 sinkpad transform srcpad
389 --------------->| | |
390 | find_transform() | |
391 |------------------->| |
393 | |--------------------->|
394 return | [caps unchanged] | |
395 <----------------------------------| |
397 | [caps changed] .-| |
398 | find_transform() | | |
400 <----------------------------------| |
403 In order to perform passthrough buffer-alloc or pad-alloc, we need to be able
404 to get the size of the output buffer after the transform.
406 For passthrough buffer-alloc, this is trivial: the input size equals the output
409 For the copy transform or the in-place transform we need additional function to
410 retrieve the size. There are two functions:
414 Given a caps and a size on one pad, and a caps on the other pad, calculate
415 the size of the other buffer. This function is able to perform all size
416 transforms and is the prefered method of transforming a size.
420 When the input size and output size are always a multiple of eachother
421 (audio conversion, ..) we can define a more simple get_unit_size() function.
422 The transform will use this function to get the same amount of units in the
423 source and destination buffers.
425 For performance reasons, the mapping between caps and size is kept in a cache.
431 passthrough and in-place transforms (with writable buffers) never need to
432 perform a pad-alloc on the srcpad. This means that if upstream negotiation
433 happens, the transform element will never know about it.
435 The transform element will keep therefore track of the allocation pattern of
436 the peer elements. We can see the following cases:
438 - upstream peer calls buffer-alloc on the sinkpad of the transform. In some
439 cases (see above) this call gets proxied or not.
441 - upstream peer does never call buffer-alloc.
443 We will keeps state about this allocation pattern and perform the following in
444 each case respectively:
446 - Upstream calls buffer-alloc: In passthrough and (some) in-place we proxy
447 this call onto the downstream element. If the caps are changed, we mark
448 a flag that we will require a new pad-alloc for the output of the next
451 - upstream peer does not call buffer-alloc: We always perform a pad-alloc
452 when processing buffers. We can further optimize by only looking at the
453 returned caps instead of doing a full, needless buffer copy.
459 videotestsrc ! ximagesink
461 - resizing happens because videotestsrc performs pad-alloc.
463 videotestsrc peer-alloc=0 ! ximagesink
465 - resizing cannot happen because videotestsrc never performs pad-alloc.
467 videotestsrc ! videoscale ! ximagesink
469 - videoscale is initially configured in passthrough mode, pad-alloc from
470 videotestsrc is proxied through videoscale.
471 - pad-alloc will renegotiate a new size in videotestsrc.
473 videotestsrc peer-alloc=0 ! videoscale ! ximagesink
475 - videoscale is initially configured in passthrough mode.
476 - videoscale performs pad-alloc because no buffer-alloc is called on the
478 - resizing the videosink makes videoscale perform the scaling.
483 filesrc location=~/media/moveyourfeet.mov ! decodebin !
484 ffmpegcolorspace ! videoscale ! ffmpegcolorspace ! ximagesink -v
projects / platform / upstream / gstreamer.git / blob