3 Transform elements transform input buffers to output buffers based on
4 the sink and source caps.
6 An important requirement for a transform is that the output caps are
7 completely defined by the input caps and vice versa. This means that a
8 typical decoder element can NOT be implemented with a transform element,
9 this is because the output caps like width and height of the
10 decompressed video frame, for example, are encoded in the stream and
11 thus not defined by the input caps.
13 Typical transform elements include:
15 - audio convertors (audioconvert, audioresample,…)
17 - video convertors (colorspace, videoscale, …)
19 - filters (capsfilter, volume, colorbalance, …)
21 The implementation of the transform element has to take care of the
24 - efficient negotiation both up and downstream
26 - efficient buffer alloc and other buffer management
28 Some transform elements can operate in different modes:
30 - passthrough (no changes are done on the input buffers)
32 - in-place (changes made directly to the incoming buffers without
33 requiring a copy or new buffer allocation)
35 - metadata changes only
37 Depending on the mode of operation the buffer allocation strategy might
40 The transform element should at any point be able to renegotiate sink
41 and src caps as well as change the operation mode.
43 In addition, the transform element will typically take care of the
44 following things as well:
50 - timestamping, this is typically done by copying the input timestamps
51 to the output buffers but subclasses should be able to override
54 - QoS, avoiding calls to the subclass transform function
56 - handle scheduling issues such as push and pull based operation.
58 In the next sections, we will describe the behaviour of the transform
59 element in each of the above use cases. We focus mostly on the buffer
60 allocation strategies and caps negotiation.
64 A transform has 2 main processing functions:
66 - **`transform()`**: Transform the input buffer to the output buffer. The
67 output buffer is guaranteed to be writable and different from the input buffer.
69 - **`transform_ip()`**: Transform the input buffer in-place. The input buffer
70 is writable and of bigger or equal size than the output buffer.
72 A transform can operate in the following modes:
74 - *passthrough*: The element will not make changes to the buffers, buffers are
75 pushed straight through, caps on both sides need to be the same. The element
76 can optionally implement a `transform_ip()` function to take a look at the data,
77 the buffer does not have to be writable.
79 - *in-place*: Changes can be made to the input buffer directly to obtain the
80 output buffer. The transform must implement a `transform_ip()` function.
82 - *copy-transform*: The transform is performed by copying and transforming the
83 input buffer to a new output buffer. The transform must implement a `transform()` function.
85 When no `transform()` function is provided, only in-place and passthrough
86 operation is allowed, this means that source and destination caps must
87 be equal or that the source buffer size is bigger or equal than the
90 When no `transform_ip()` function is provided, only passthrough and
91 copy-transforms are supported. Providing this function is an
92 optimisation that can avoid a buffer copy.
94 When no functions are provided, we can only process in passthrough mode.
98 Typical (re)negotiation of the transform element in push mode always
99 goes from sink to src, this means triggers the following sequence:
101 - the sinkpad receives a new caps event.
103 - the transform function figures out what it can convert these caps
106 - try to see if we can configure the caps unmodified on the peer. We
107 need to do this because we prefer to not do anything.
109 - the transform configures itself to transform from the new sink caps
110 to the target src caps
112 - the transform processes and sets the output caps on the src pad
114 We call this downstream negotiation (DN) and it goes roughly like this:
117 sinkpad transform srcpad
119 ------------>| find_transform() | |
120 |------------------->| |
122 | |--------------------->|
123 | <configure caps> <-| |
126 These steps configure the element for a transformation from the input
127 caps to the output caps.
129 The transform has 3 function to perform the negotiation:
131 - **`transform_caps()`**: Transform the caps on a certain pad to all the
132 possible supported caps on the other pad. The input caps are guaranteed to be
133 a simple caps with just one structure. The caps do not have to be fixed.
135 - **`fixate_caps()`**: Given a caps on one pad, fixate the caps on the other
136 pad. The target caps are writable.
138 - **`set_caps()`**: Configure the transform for a transformation between src
139 caps and dest caps. Both caps are guaranteed to be fixed caps.
141 If no `transform_caps()` is defined, we can only perform the identity
142 transform, by default.
144 If no `set_caps()` is defined, we don’t care about caps. In that case we
145 also assume nothing is going to write to the buffer and we don’t enforce
146 a writable buffer for the `transform_ip()` function, when present.
148 One common function that we need for the transform element is to find
149 the best transform from one format (src) to another (dest). Some
150 requirements of this function are:
152 - has a fixed src caps
154 - finds a fixed dest caps that the transform element can transform to
156 - the dest caps are compatible and can be accepted by peer elements
158 - the transform function prefers to make src caps == dest caps
160 - the transform function can optionally fixate dest caps.
162 The `find_transform()` function goes like this:
164 - start from src aps, these caps are fixed.
166 - check if the caps are acceptable for us as src caps. This is usually
167 enforced by the padtemplate of the element.
169 - calculate all caps we can transform too with `transform_caps()`
171 - if the original caps are a subset of the transforms, try to see if
172 the the caps are acceptable for the peer. If this is possible, we
173 can perform passthrough and make src == dest. This is performed by
174 simply calling `gst_pad_peer_query_accept_caps()`.
176 - if the caps are not fixed, we need to fixate it, start by taking the
177 peer caps and intersect with them.
179 - for each of the transformed caps retrieved with `transform_caps()`:
181 - try to fixate the caps with `fixate_caps()`
183 - if the caps are fixated, check if the peer accepts them with
184 `_peer_query_accept_caps()`, if the peer accepts, we have found a dest caps.
186 - if we run out of caps, we fail to find a transform.
188 - if we found a destination caps, configure the transform with
191 After this negotiation process, the transform element is usually in a
192 steady state. We can identify these steady states:
194 - src and sink pads both have the same caps. Note that when the caps
195 are equal on both pads, the input and output buffers automatically
196 have the same size. The element can operate on the buffers in the
197 following ways: (Same caps, SC)
199 - passthrough: buffers are inspected but no metadata or buffer data is
200 changed. The input buffers don’t need to be writable. The input
201 buffer is simply pushed out again without modifications. (SCP)
204 sinkpad transform srcpad
206 ------------>| handle_buffer() | |
207 |------------------->| pad_push() |
208 | |--------------------->|
212 - in-place: buffers are modified in-place, this means that the input
213 buffer is modified to produce a new output buffer. This requires the
214 input buffer to be writable. If the input buffer is not writable, a
215 new buffer has to be allocated from the bufferpool. (SCI)
218 sinkpad transform srcpad
220 ------------>| handle_buffer() | |
221 |------------------->| |
225 | <transform_ip> | | |
228 | |--------------------->|
232 - copy transform: a new output buffer is allocate from the bufferpool
233 and data from the input buffer is transformed into the output
237 sinkpad transform srcpad
239 ------------>| handle_buffer() | |
240 |------------------->| |
246 | |--------------------->|
250 - src and sink pads have different caps. The element can operate on
251 the buffers in the following way: (Different Caps, DC)
253 - in-place: input buffers are modified in-place. This means that the
254 input buffer has a size that is larger or equal to the output size.
255 The input buffer will be resized to the size of the output buffer.
256 If the input buffer is not writable or the output size is bigger
257 than the input size, we need to pad-alloc a new buffer. (DCI)
260 sinkpad transform srcpad
262 ------------>| handle_buffer() | |
263 |------------------->| |
264 | | [!writable || !size] |
267 | <transform_ip> | | |
270 | |--------------------->|
274 - copy transform: a new output buffer is allocated and the data from
275 the input buffer is transformed into the output buffer. The flow is
276 exactly the same as the case with the same-caps negotiation. (DCC)
278 We can immediately observe that the copy transform states will need to
279 allocate a new buffer from the bufferpool. When the transform element is
280 receiving a non-writable buffer in the in-place state, it will also need
281 to perform an allocation. There is no reason why the passthrough state
282 would perform an allocation.
284 This steady state changes when one of the following actions occur:
286 - the sink pad receives new caps, this triggers the above downstream
287 renegotation process, see above for the flow.
289 - the transform element wants to renegotiate (because of changed
290 properties, for example). This essentially clears the current steady
291 state and triggers the downstream and upstream renegotiation
292 process. This situation also happens when a RECONFIGURE event was
293 received on the transform srcpad.
297 After the transform element is configured with caps, a bufferpool needs
298 to be negotiated to perform the allocation of buffers. We have 2 cases:
300 - The element is operating in passthrough we don’t need to allocate a
301 buffer in the transform element.
303 - The element is not operating in passthrough and needs to allocation
306 In case 1, we don’t query and configure a pool. We let upstream decide
307 if it wants to use a bufferpool and then we will proxy the bufferpool
308 from downstream to upstream.
310 In case 2, we query and set a bufferpool on the srcpad that will be used
311 for doing the allocations.
313 In order to perform allocation, we need to be able to get the size of
314 the output buffer after the transform. We need additional function to
315 retrieve the size. There are two functions:
317 - `transform_size()`: Given a caps and a size on one pad, and a caps on the
318 other pad, calculate the size of the other buffer. This function is able to
319 perform all size transforms and is the preferred method of transforming
322 - `get_unit_size()`: When the input size and output size are always
323 a multiple of each other (audio conversion, ..) we can define a more simple
324 `get_unit_size()` function. The transform will use this function to get the
325 same amount of units in the source and destination buffers. For performance
326 reasons, the mapping between caps and size is kept in a cache.