1 <chapter id="chapter-autoplugging">
2 <title>Autoplugging</title>
4 In <xref linkend="chapter-helloworld"/>, you've learned to build a
5 simple media player for Ogg/Vorbis files. By using alternative elements,
6 you are able to build media players for other media types, such as
7 Ogg/Speex, MP3 or even video formats. However, you would rather want
8 to build an application that can automatically detect the media type
9 of a stream and automatically generate the best possible pipeline
10 by looking at all available elements in a system. This process is called
11 autoplugging, and &GStreamer; contains high-quality autopluggers. If
12 you're looking for an autoplugger, don't read any further and go to
13 <xref linkend="chapter-components"/>. This chapter will explain the
14 <emphasis>concept</emphasis> of autoplugging and typefinding. It will
15 explain what systems &GStreamer; includes to dynamically detect the
16 type of a media stream, and how to generate a pipeline of decoder
17 elements to playback this media. The same principles can also be used
18 for transcoding. Because of the full dynamicity of this concept,
19 &GStreamer; can be automatically extended to support new media types
20 without needing any adaptations to its autopluggers.
23 We will first introduce the concept of MIME types as a dynamic and
24 extendible way of identifying media streams. After that, we will introduce
25 the concept of typefinding to find the type of a media stream. Lastly,
26 we will explain how autoplugging and the &GStreamer; registry can be
27 used to setup a pipeline that will convert media from one mimetype to
28 another, for example for media decoding.
31 <sect1 id="section-mime">
32 <title>MIME-types as a way to identify streams</title>
34 We have previously introduced the concept of capabilities as a way
35 for elements (or, rather, pads) to agree on a media type when
36 streaming data from one element to the next (see <xref
37 linkend="section-caps"/>). We have explained that a capability is
38 a combination of a mimetype and a set of properties. For most
39 container formats (those are the files that you will find on your
40 hard disk; Ogg, for example, is a container format), no properties
41 are needed to describe the stream. Only a MIME-type is needed. A
42 full list of MIME-types and accompanying properties can be found
44 url="http://gstreamer.freedesktop.org/data/doc/gstreamer/head/pwg/html/section-types-definitions.html">the
45 Plugin Writer's Guide</ulink>.
48 An element must associate a MIME-type to its source and sink pads
49 when it is loaded into the system. &GStreamer; knows about the
50 different elements and what type of data they expect and emit through
51 the &GStreamer; registry. This allows for very dynamic and extensible
52 element creation as we will see.
56 In <xref linkend="chapter-helloworld"/>, we've learned to build a
57 music player for Ogg/Vorbis files. Let's look at the MIME-types
58 associated with each pad in this pipeline. <xref
59 linkend="section-mime-img"/> shows what MIME-type belongs to each
63 <figure float="1" id="section-mime-img">
64 <title>The Hello world pipeline with MIME types</title>
67 <imagedata scale="75" fileref="images/mime-world.ℑ" format="&IMAGE;"/>
73 Now that we have an idea how &GStreamer; identifies known media
74 streams, we can look at methods &GStreamer; uses to setup pipelines
75 for media handling and for media type detection.
79 <sect1 id="section-typefinding">
80 <title>Media stream type detection</title>
82 Usually, when loading a media stream, the type of the stream is not
83 known. This means that before we can choose a pipeline to decode the
84 stream, we first need to detect the stream type. &GStreamer; uses the
85 concept of typefinding for this. Typefinding is a normal part of a
86 pipeline, it will read data for as long as the type of a stream is
87 unknown. During this period, it will provide data to all plugins
88 that implement a typefinder. When one of the typefinders recognizes
89 the stream, the typefind element will emit a signal and act as a
90 passthrough module from that point on. If no type was found, it will
91 emit an error and further media processing will stop.
94 Once the typefind element has found a type, the application can
95 use this to plug together a pipeline to decode the media stream.
96 This will be discussed in the next section.
99 Plugins in &GStreamer; can, as mentioned before, implement typefinder
100 functionality. A plugin implementing this functionality will submit
101 a mimetype, optionally a set of file extensions commonly used for this
102 media type, and a typefind function. Once this typefind function inside
103 the plugin is called, the plugin will see if the data in this media
104 stream matches a specific pattern that marks the media type identified
105 by that mimetype. If it does, it will notify the typefind element of
106 this fact, telling which mediatype was recognized and how certain we
107 are that this stream is indeed that mediatype. Once this run has been
108 completed for all plugins implementing a typefind functionality, the
109 typefind element will tell the application what kind of media stream
110 it thinks to have recognized.
113 The following code should explain how to use the typefind element.
114 It will print the detected media type, or tell that the media type
115 was not found. The next section will introduce more useful behaviours,
116 such as plugging together a decoding pipeline.
118 <programlisting><!-- example-begin typefind.c a -->
119 #include <gst/gst.h>
120 <!-- example-end typefind.c a -->
121 [.. my_bus_callback goes here ..]<!-- example-begin typefind.c b --><!--
123 my_bus_callback (GstBus *bus,
127 GMainLoop *loop = data;
129 switch (GST_MESSAGE_TYPE (message)) {
130 case GST_MESSAGE_ERROR: {
134 gst_message_parse_error (message, &err, &debug);
135 g_print ("Error: %s\n", err->message);
139 g_main_loop_quit (loop);
142 case GST_MESSAGE_EOS:
144 g_main_loop_quit (loop);
150 /* remove from queue */
153 --><!-- example-end typefind.c b -->
154 <!-- example-begin typefind.c c -->
156 idle_exit_loop (gpointer data)
158 g_main_loop_quit ((GMainLoop *) data);
165 cb_typefound (GstElement *typefind,
170 GMainLoop *loop = data;
173 type = gst_caps_to_string (caps);
174 g_print ("Media type %s found, probability %d%%\n", type, probability);
177 /* since we connect to a signal in the pipeline thread context, we need
178 * to set an idle handler to exit the main loop in the mainloop context.
179 * Normally, your app should not need to worry about such things. */
180 g_idle_add (idle_exit_loop, loop);
188 GstElement *pipeline, *filesrc, *typefind, *fakesink;
192 gst_init (&argc, &argv);
193 loop = g_main_loop_new (NULL, FALSE);
197 g_print ("Usage: %s <filename>\n", argv[0]);
201 /* create a new pipeline to hold the elements */
202 pipeline = gst_pipeline_new ("pipe");
204 bus = gst_pipeline_get_bus (GST_PIPELINE (pipeline));
205 gst_bus_add_watch (bus, my_bus_callback, NULL);
206 gst_object_unref (bus);
208 /* create file source and typefind element */
209 filesrc = gst_element_factory_make ("filesrc", "source");
210 g_object_set (G_OBJECT (filesrc), "location", argv[1], NULL);
211 typefind = gst_element_factory_make ("typefind", "typefinder");
212 g_signal_connect (typefind, "have-type", G_CALLBACK (cb_typefound), loop);
213 fakesink = gst_element_factory_make ("fakesink", "sink");
216 gst_bin_add_many (GST_BIN (pipeline), filesrc, typefind, fakesink, NULL);
217 gst_element_link_many (filesrc, typefind, fakesink, NULL);
218 gst_element_set_state (GST_ELEMENT (pipeline), GST_STATE_PLAYING);
219 g_main_loop_run (loop);
222 gst_element_set_state (GST_ELEMENT (pipeline), GST_STATE_NULL);
223 gst_object_unref (GST_OBJECT (pipeline));
227 <!-- example-end typefind.c c --></programlisting>
229 Once a media type has been detected, you can plug an element (e.g. a
230 demuxer or decoder) to the source pad of the typefind element, and
231 decoding of the media stream will start right after.
235 <sect1 id="section-dynamic">
236 <title>Plugging together dynamic pipelines</title>
238 The code in this section is broken, outdated and overly complicated.
239 Also, you should use decodebin, playbin or uridecodebin to get
240 decoders plugged automatically.
243 In this chapter we will see how you can create a dynamic pipeline. A
244 dynamic pipeline is a pipeline that is updated or created while data
245 is flowing through it. We will create a partial pipeline first and add
246 more elements while the pipeline is playing. The basis of this player
247 will be the application that we wrote in the previous section (<xref
248 linkend="section-typefinding"/>) to identify unknown media streams.
250 <!-- example-begin dynamic.c a --><!--
251 #include <gst/gst.h>
253 GstElement *pipeline;
254 --><!-- example-end dynamic.c a -->
256 Once the type of the media has been found, we will find elements in
257 the registry that can decode this streamtype. For this, we will get
258 all element factories (which we've seen before in <xref
259 linkend="section-elements-create"/>) and find the ones with the
260 given MIME-type and capabilities on their sinkpad. Note that we will
261 only use parsers, demuxers and decoders. We will not use factories for
262 any other element types, or we might get into a loop of encoders and
263 decoders. For this, we will want to build a list of <quote>allowed</quote>
264 factories right after initializing &GStreamer;.
266 <programlisting><!-- example-begin dynamic.c b -->
267 static GList *factories;
270 * This function is called by the registry loader. Its return value
271 * (TRUE or FALSE) decides whether the given feature will be included
272 * in the list that we're generating further down.
276 cb_feature_filter (GstPluginFeature *feature,
282 /* we only care about element factories */
283 if (!GST_IS_ELEMENT_FACTORY (feature))
286 /* only parsers, demuxers and decoders */
287 klass = gst_element_factory_get_klass (GST_ELEMENT_FACTORY (feature));
288 if (g_strrstr (klass, "Demux") == NULL &&
289 g_strrstr (klass, "Decoder") == NULL &&
290 g_strrstr (klass, "Parse") == NULL)
293 /* only select elements with autoplugging rank */
294 rank = gst_plugin_feature_get_rank (feature);
295 if (rank < GST_RANK_MARGINAL)
302 * This function is called to sort features by rank.
306 cb_compare_ranks (GstPluginFeature *f1,
307 GstPluginFeature *f2)
309 return gst_plugin_feature_get_rank (f2) - gst_plugin_feature_get_rank (f1);
313 init_factories (void)
315 /* first filter out the interesting element factories */
316 factories = gst_registry_feature_filter (
317 gst_registry_get_default (),
318 (GstPluginFeatureFilter) cb_feature_filter, FALSE, NULL);
320 /* sort them according to their ranks */
321 factories = g_list_sort (factories, (GCompareFunc) cb_compare_ranks);
323 <!-- example-end dynamic.c b --></programlisting>
325 From this list of element factories, we will select the one that most
326 likely will help us decoding a media stream to a given output type.
327 For each newly created element, we will again try to autoplug new
328 elements to its source pad(s). Also, if the element has dynamic pads
329 (which we've seen before in <xref linkend="section-pads-dynamic"/>),
330 we will listen for newly created source pads and handle those, too.
331 The following code replaces the <function>cb_type_found</function>
332 from the previous section with a function to initiate autoplugging,
333 which will continue with the above approach.
335 <programlisting><!-- example-begin dynamic.c c -->
336 static void try_to_plug (GstPad *pad, const GstCaps *caps);
338 static GstElement *audiosink;
341 cb_newpad (GstElement *element,
347 caps = gst_pad_get_caps (pad, NULL);
348 try_to_plug (pad, caps);
349 gst_caps_unref (caps);
353 close_link (GstPad *srcpad,
354 GstElement *sinkelement,
355 const gchar *padname,
356 const GList *templlist)
359 gboolean has_dynamic_pads = FALSE;
361 g_print ("Plugging pad %s:%s to newly created %s:%s\n",
362 gst_object_get_name (GST_OBJECT (gst_pad_get_parent (srcpad))),
363 gst_pad_get_name (srcpad),
364 gst_object_get_name (GST_OBJECT (sinkelement)), padname);
366 /* add the element to the pipeline and set correct state */
367 if (sinkelement != audiosink) {
368 gst_bin_add (GST_BIN (pipeline), sinkelement);
369 gst_element_set_state (sinkelement, GST_STATE_READY);
371 pad = gst_element_get_static_pad (sinkelement, padname);
372 gst_pad_link (srcpad, pad);
373 if (sinkelement != audiosink) {
374 gst_element_set_state (sinkelement, GST_STATE_PAUSED);
376 gst_object_unref (GST_OBJECT (pad));
378 /* if we have static source pads, link those. If we have dynamic
379 * source pads, listen for pad-added signals on the element */
380 for ( ; templlist != NULL; templlist = templlist->next) {
381 GstStaticPadTemplate *templ = templlist->data;
383 /* only sourcepads, no request pads */
384 if (templ->direction != GST_PAD_SRC ||
385 templ->presence == GST_PAD_REQUEST) {
389 switch (templ->presence) {
390 case GST_PAD_ALWAYS: {
391 GstPad *pad = gst_element_get_static_pad (sinkelement, templ->name_template);
392 GstCaps *caps = gst_pad_get_caps (pad, NULL);
395 try_to_plug (pad, caps);
396 gst_object_unref (GST_OBJECT (pad));
397 gst_caps_unref (caps);
400 case GST_PAD_SOMETIMES:
401 has_dynamic_pads = TRUE;
408 /* listen for newly created pads if this element supports that */
409 if (has_dynamic_pads) {
410 g_signal_connect (sinkelement, "pad-added", G_CALLBACK (cb_newpad), NULL);
415 try_to_plug (GstPad *pad,
418 GstObject *parent = GST_OBJECT (GST_OBJECT_PARENT (pad));
421 GstCaps *res, *audiocaps;
423 /* don't plug if we're already plugged - FIXME: memleak for pad */
424 if (GST_PAD_IS_LINKED (gst_element_get_static_pad (audiosink, "sink"))) {
425 g_print ("Omitting link for pad %s:%s because we're already linked\n",
426 GST_OBJECT_NAME (parent), GST_OBJECT_NAME (pad));
430 /* as said above, we only try to plug audio... Omit video */
431 mime = gst_structure_get_name (gst_caps_get_structure (caps, 0));
432 if (g_strrstr (mime, "video")) {
433 g_print ("Omitting link for pad %s:%s because mimetype %s is non-audio\n",
434 GST_OBJECT_NAME (parent), GST_OBJECT_NAME (pad), mime);
438 /* can it link to the audiopad? */
439 audiocaps = gst_pad_get_caps (gst_element_get_static_pad (audiosink, "sink"),
441 res = gst_caps_intersect (caps, audiocaps);
442 if (res && !gst_caps_is_empty (res)) {
443 g_print ("Found pad to link to audiosink - plugging is now done\n");
444 close_link (pad, audiosink, "sink", NULL);
445 gst_caps_unref (audiocaps);
446 gst_caps_unref (res);
449 gst_caps_unref (audiocaps);
450 gst_caps_unref (res);
452 /* try to plug from our list */
453 for (item = factories; item != NULL; item = item->next) {
454 GstElementFactory *factory = GST_ELEMENT_FACTORY (item->data);
457 for (pads = gst_element_factory_get_static_pad_templates (factory);
458 pads != NULL; pads = pads->next) {
459 GstStaticPadTemplate *templ = pads->data;
461 /* find the sink template - need an always pad*/
462 if (templ->direction != GST_PAD_SINK ||
463 templ->presence != GST_PAD_ALWAYS) {
468 res = gst_caps_intersect (caps,
469 gst_static_caps_get (&templ->static_caps));
470 if (res && !gst_caps_is_empty (res)) {
472 gchar *name_template = g_strdup (templ->name_template);
474 /* close link and return */
475 gst_caps_unref (res);
476 element = gst_element_factory_create (factory, NULL);
477 close_link (pad, element, name_template,
478 gst_element_factory_get_static_pad_templates (factory));
479 g_free (name_template);
482 gst_caps_unref (res);
484 /* we only check one sink template per factory, so move on to the
485 * next factory now */
490 /* if we get here, no item was found */
491 g_print ("No compatible pad found to decode %s on %s:%s\n",
492 mime, GST_OBJECT_NAME (parent), GST_OBJECT_NAME (pad));
496 cb_typefound (GstElement *typefind,
504 s = gst_caps_to_string (caps);
505 g_print ("Detected media type %s\n", s);
508 /* actually plug now */
509 pad = gst_element_get_static_pad (typefind, "src");
510 try_to_plug (pad, caps);
511 gst_object_unref (GST_OBJECT (pad));
513 <!-- example-end dynamic.c c --></programlisting>
515 By doing all this, we will be able to make a simple autoplugger that
516 can automatically setup a pipeline for any media type. In the example
517 above, we did this for audio only. However, we can also do this
518 for video to create a player that plays both audio and video.
520 <!-- example-begin dynamic.c d --><!--
522 my_bus_callback (GstBus *bus,
526 GMainLoop *loop = data;
528 switch (GST_MESSAGE_TYPE (message)) {
529 case GST_MESSAGE_ERROR: {
533 gst_message_parse_error (message, &err, &debug);
534 g_print ("Error: %s\n", err->message);
538 g_main_loop_quit (loop);
541 case GST_MESSAGE_EOS:
543 g_main_loop_quit (loop);
549 /* remove from queue */
558 GstElement *typefind, *realsink;
563 /* init GStreamer and ourselves */
564 gst_init (&argc, &argv);
565 loop = g_main_loop_new (NULL, FALSE);
570 g_print ("Usage: %s <filename>\n", argv[0]);
575 p = g_strdup_printf ("filesrc location=\"%s\" ! typefind name=tf", argv[1]);
576 pipeline = gst_parse_launch (p, &err);
580 g_error ("Could not construct pipeline: %s", err->message);
585 bus = gst_pipeline_get_bus (GST_PIPELINE (pipeline));
586 gst_bus_add_watch (bus, my_bus_callback, NULL);
587 gst_object_unref (bus);
589 typefind = gst_bin_get_by_name (GST_BIN (pipeline), "tf");
590 g_signal_connect (typefind, "have-type", G_CALLBACK (cb_typefound), NULL);
591 gst_object_unref (GST_OBJECT (typefind));
592 audiosink = gst_element_factory_make ("audioconvert", "aconv");
593 realsink = gst_element_factory_make ("alsasink", "audiosink");
594 gst_bin_add_many (GST_BIN (pipeline), audiosink, realsink, NULL);
595 gst_element_link (audiosink, realsink);
596 gst_element_set_state (pipeline, GST_STATE_PLAYING);
599 g_main_loop_run (loop);
602 gst_element_set_state (pipeline, GST_STATE_NULL);
603 gst_object_unref (GST_OBJECT (pipeline));
607 --><!-- example-end dynamic.c d -->
609 The example above is a good first try for an autoplugger. Next steps
610 would be to listen for <quote>pad-removed</quote> signals, so we
611 can dynamically change the plugged pipeline if the stream changes
612 (this happens for DVB or Ogg radio). Also, you might want special-case
613 code for input with known content (such as a DVD or an audio-CD),
614 and much, much more. Moreover, you'll want many checks to prevent
615 infinite loops during autoplugging, maybe you'll want to implement
616 shortest-path-finding to make sure the most optimal pipeline is chosen,
617 and so on. Basically, the features that you implement in an autoplugger
618 depend on what you want to use it for. For full-blown implementations,
619 see the <quote>playbin</quote> and <quote>decodebin</quote> elements in
620 <xref linkend="chapter-components"/>.