8 Autoplug and decode to raw media
10 Input : single pad with ANY caps Output : Dynamic pads
14 _ a GstTypeFindElement connected to the single sink pad
16 _ optionally a demuxer/parser
18 _ optionally one or more DecodeGroup
22 The goal is to reach 'target' caps (by default raw media).
24 This is done by using the GstCaps of a source pad and finding the available
25 demuxers/decoders GstElement that can be linked to that pad.
27 The process starts with the source pad of typefind and stops when no more
28 non-target caps are left. It is commonly done while pre-rolling, but can also
29 happen whenever a new pad appears on any element.
31 Once a target caps has been found, that pad is ghosted and the
32 'new-decoded-pad' signal is emitted.
34 If no compatible elements can be found for a GstCaps, the pad is ghosted and
35 the 'unknown-type' signal is emitted.
38 * Assisted auto-plugging
40 When starting the auto-plugging process for a given GstCaps, two signals are
41 emitted in the following way in order to allow the application/user to assist or
42 fine-tune the process.
44 _ 'autoplug-continue' :
46 gboolean user_function (GstElement * decodebin, GstPad *pad, GstCaps * caps)
48 This signal is fired at the very beginning with the source pad GstCaps. If
49 the callback returns TRUE, the process continues normally. If the callback
50 returns FALSE, then the GstCaps are considered as a target caps and the
51 autoplugging process stops.
53 - 'autoplug-factories' :
55 GValueArray user_function (GstElement* decodebin, GstPad* pad,
58 Get a list of elementfactories for @pad with @caps. This function is used to
59 instruct decodebin2 of the elements it should try to autoplug. The default
60 behaviour when this function is not overriden is to get all elements that
61 can handle @caps from the registry sorted by rank.
65 gint user_function (GstElement* decodebin, GstPad* pad, GstCaps* caps,
66 GValueArray* factories);
68 This signal is fired once autoplugging has got a list of compatible
69 GstElementFactory. The signal is emitted with the GstCaps of the source pad
70 and a pointer on the GValueArray of compatible factories.
72 The callback should return the index of the elementfactory in @factories
73 that should be tried next.
75 If the callback returns -1, the autoplugging process will stop as if no
76 compatible factories were found.
78 The default implementation of this function will try to autoplug the first
83 The target caps are a read/write GObject property of decodebin.
85 By default the target caps are:
87 _ Raw audio : audio/x-raw-int, audio/x-raw-float
89 _ and raw video : video/x-raw-rgb, video/x-raw-yuv
91 _ and Text : text/plain, text/x-pango-markup
94 * media chain/group handling
96 When autoplugging, all streams coming out of a demuxer will be grouped in a
99 All new source pads created on that demuxer after it has emitted the
100 'no-more-pads' signal will be put in another DecodeGroup.
102 Only one decodegroup can be active at any given time. If a new decodegroup is
103 created while another one exists, that decodegroup will be set as blocking until
104 the existing one has drained.
113 Streams belonging to the same group/chain of a media file.
117 The DecodeGroup contains:
119 _ a GstMultiQueue to which all streams of a the media group are connected.
121 _ the eventual decoders which are autoplugged in order to produce the
122 requested target pads.
124 * Proper group draining
126 The DecodeGroup takes care that all the streams in the group are completely
127 drained (EOS has come through all source ghost pads).
131 The DecodeGroup has a global blocking feature. If enabled, all the ghosted
132 source pads for that group will be blocked.
134 A method is available to unblock all blocked pads for that group.
143 Multiple input-output data queue
145 The GstMultiQueue achieves the same functionality as GstQueue, with a few
148 * Multiple streams handling.
150 The element handles queueing data on more than one stream at once. To
151 achieve such a feature it has request sink pads (sink_%d) and 'sometimes' src
154 When requesting a given sinkpad, the associated srcpad for that stream will
155 be created. Ex: requesting sink_1 will generate src_1.
158 * Non-starvation on multiple streams.
160 If more than one stream is used with the element, the streams' queues will
161 be dynamically grown (up to a limit), in order to ensure that no stream is
162 risking data starvation. This guarantees that at any given time there are at
163 least N bytes queued and available for each individual stream.
165 If an EOS event comes through a srcpad, the associated queue should be
166 considered as 'not-empty' in the queue-size-growing algorithm.
169 * Non-linked srcpads graceful handling.
171 A GstTask is started for all srcpads when going to GST_STATE_PAUSED.
173 The task are blocking against a GCondition which will be fired in two
176 _ When the associated queue has received a buffer.
178 _ When the associated queue was previously declared as 'not-linked' and the
179 first buffer of the queue is scheduled to be pushed synchronously in
180 relation to the order in which it arrived globally in the element (see
181 'Synchronous data pushing' below).
183 When woken up by the GCondition, the GstTask will try to push the next
184 GstBuffer/GstEvent on the queue. If pushing the GstBuffer/GstEvent returns
185 GST_FLOW_NOT_LINKED, then the associated queue is marked as 'not-linked'. If
186 pushing the GstBuffer/GstEvent succeeded the queue will no longer be marked as
189 If pushing on all srcpads returns GstFlowReturn different from GST_FLOW_OK,
190 then all the srcpads' tasks are stopped and subsequent pushes on sinkpads will
191 return GST_FLOW_NOT_LINKED.
193 * Synchronous data pushing for non-linked pads.
195 In order to better support dynamic switching between streams, the multiqueue
196 (unlike the current GStreamer queue) continues to push buffers on non-linked
197 pads rather than shutting down.
199 In addition, to prevent a non-linked stream from very quickly consuming all
200 available buffers and thus 'racing ahead' of the other streams, the element
201 must ensure that buffers and inlined events for a non-linked stream are pushed
202 in the same order as they were received, relative to the other streams
203 controlled by the element. This means that a buffer cannot be pushed to a
204 non-linked pad any sooner than buffers in any other stream which were received
208 =====================================
209 Parsers, decoders and auto-plugging
210 =====================================
212 This section has DRAFT status.
214 Some media formats come in different "flavours" or "stream formats". These
215 formats differ in the way the setup data and media data is signalled and/or
216 packaged. An example for this is H.264 video, where there is a bytestream
217 format (with codec setup data signalled inline and units prefixed by a sync
218 code and packet length information) and a "raw" format where codec setup
219 data is signalled out of band (via the caps) and the chunking is implicit
220 in the way the buffers were muxed into a container, to mention just two of
221 the possible variants.
223 Especially on embedded platforms it is common that decoders can only
224 handle one particular stream format, and not all of them.
226 Where there are multiple stream formats, parsers are usually expected
227 to be able to convert between the different formats. This will, if
228 implemented correctly, work as expected in a static pipeline such as
230 ... ! parser ! decoder ! sink
232 where the parser can query the decoder's capabilities even before
233 processing the first piece of data, and configure itself to convert
234 accordingly, if conversion is needed at all.
236 In an auto-plugging context this is not so straight-forward though,
237 because elements are plugged incrementally and not before the previous
238 element has processes some data and decided what it will output exactly
239 (unless the template caps are completely fixed, then it can continue
240 right away, this is not always the case here though, see below). A
241 parser will thus have to decide on *some* output format so auto-plugging
242 can continue. It doesn't know anything about the available decoders and
243 their capabilities though, so it's possible that it will choose a format
244 that is not supported by any of the available decoders, or by the preferred
247 If the parser had sufficiently concise but fixed source pad template caps,
248 decodebin could continue to plug a decoder right away, allowing the
249 parser to configure itself in the same way as it would with a static
250 pipeline. This is not an option, unfortunately, because often the
251 parser needs to process some data to determine e.g. the format's profile or
252 other stream properties (resolution, sample rate, channel configuration, etc.),
253 and there may be different decoders for different profiles (e.g. DSP codec
254 for baseline profile, and software fallback for main/high profile; or a DSP
255 codec only supporting certain resolutions, with a software fallback for
256 unusual resolutions). So if decodebin just plugged the most highest-ranking
257 decoder, that decoder might not be be able to handle the actual stream later
258 on, which would yield an error (this is a data flow error then which would
259 be hard to intercept and avoid in decodebin). In other words, we can't solve
260 this issue by plugging a decoder right away with the parser.
262 So decodebin needs to communicate to the parser the set of available decoder
263 caps (which would contain the relevant capabilities/restrictions such as
264 supported profiles, resolutions, etc.), after the usual "autoplug-*" signal
265 filtering/sorting of course.
267 This is done by plugging a capsfilter element right after the parser, and
268 constructing set of filter caps from the list of available decoders (one
269 appends at the end just the name(s) of the caps structures from the parser
270 pad template caps to function as an 'ANY other' caps equivalent). This let
271 the parser negotiate to a supported stream format in the same way as with
272 the static pipeline mentioned above, but of course incur some overhead
273 through the additional capsfilter element.