1 <chapter id="chapter-dataaccess">
2 <title>Pipeline manipulation</title>
4 This chapter will discuss how you can manipulate your pipeline in several
5 ways from your application on. Parts of this chapter are downright
6 hackish, so be assured that you'll need some programming knowledge
7 before you start reading this.
10 Topics that will be discussed here include how you can insert data into
11 a pipeline from your application, how to read data from a pipeline,
12 how to manipulate the pipeline's speed, length, starting point and how
13 to listen to a pipeline's data processing.
16 <sect1 id="section-data-probe">
17 <title>Data probing</title>
19 Probing is best envisioned as a pad listener. Technically, a probe is
20 nothing more than a callback that can be attached to a pad.
21 You can attach a probe using <function>gst_pad_add_probe ()</function>.
22 Similarly, one can use the
23 <function>gst_pad_remove_probe ()</function>
24 to remove the callback again.
27 Probes run in pipeline threading context, so callbacks should try to
28 not block and generally not do any weird stuff, since this could
29 have a negative impact on pipeline performance or, in case of bugs,
30 cause deadlocks or crashes. More precisely, one should usually not
31 call any GUI-related functions from within a probe callback, nor try
32 to change the state of the pipeline. An application may post custom
33 messages on the pipeline's bus though to communicate with the main
34 application thread and have it do things like stop the pipeline.
37 In any case, most common buffer operations
38 that elements can do in <function>_chain ()</function> functions, can
39 be done in probe callbacks as well. The example below gives a short
40 impression on how to use them (even if this usage is not entirely
41 correct, but more on that below):
43 <programlisting><!-- example-begin probe.c -->
44 #include <gst/gst.h>
46 static GstPadProbeReturn
47 cb_have_data (GstPad *pad,
48 GstPadProbeInfo *info,
56 buffer = GST_PAD_PROBE_INFO_BUFFER (info);
58 buffer = gst_buffer_make_writable (buffer);
60 gst_buffer_map (buffer, &map, GST_MAP_WRITE);
62 ptr = (guint16 *) map.data;
64 for (y = 0; y < 288; y++) {
65 for (x = 0; x < 384 / 2; x++) {
67 ptr[384 - 1 - x] = ptr[x];
72 gst_buffer_unmap (buffer, &map);
74 GST_PAD_PROBE_INFO_DATA (info) = buffer;
84 GstElement *pipeline, *src, *sink, *filter, *csp;
89 gst_init (&argc, &argv);
90 loop = g_main_loop_new (NULL, FALSE);
93 pipeline = gst_pipeline_new ("my-pipeline");
94 src = gst_element_factory_make ("videotestsrc", "src");
96 g_error ("Could not create 'videotestsrc' element");
98 filter = gst_element_factory_make ("capsfilter", "filter");
99 g_assert (filter != NULL); /* should always exist */
101 csp = gst_element_factory_make ("videoconvert", "csp");
103 g_error ("Could not create 'videoconvert' element");
105 sink = gst_element_factory_make ("xvimagesink", "sink");
107 sink = gst_element_factory_make ("ximagesink", "sink");
109 g_error ("Could not create neither 'xvimagesink' nor 'ximagesink' element");
112 gst_bin_add_many (GST_BIN (pipeline), src, filter, csp, sink, NULL);
113 gst_element_link_many (src, filter, csp, sink, NULL);
114 filtercaps = gst_caps_new_simple ("video/x-raw",
115 "format", G_TYPE_STRING, "RGB16",
116 "width", G_TYPE_INT, 384,
117 "height", G_TYPE_INT, 288,
118 "framerate", GST_TYPE_FRACTION, 25, 1,
120 g_object_set (G_OBJECT (filter), "caps", filtercaps, NULL);
121 gst_caps_unref (filtercaps);
123 pad = gst_element_get_static_pad (src, "src");
124 gst_pad_add_probe (pad, GST_PAD_PROBE_TYPE_BUFFER,
125 (GstPadProbeCallback) cb_have_data, NULL, NULL);
126 gst_object_unref (pad);
129 gst_element_set_state (pipeline, GST_STATE_PLAYING);
131 /* wait until it's up and running or failed */
132 if (gst_element_get_state (pipeline, NULL, NULL, -1) == GST_STATE_CHANGE_FAILURE) {
133 g_error ("Failed to go into PLAYING state");
136 g_print ("Running ...\n");
137 g_main_loop_run (loop);
140 gst_element_set_state (pipeline, GST_STATE_NULL);
141 gst_object_unref (pipeline);
145 <!-- example-end probe.c --></programlisting>
147 Compare that output with the output of <quote>gst-launch-1.0
148 videotestsrc ! xvimagesink</quote>, just so you know what you're
152 Strictly speaking, a pad probe callback is only allowed to modify the
153 buffer content if the buffer is writable. Whether this is the case or
154 not depends a lot on the pipeline and the elements involved. Often
155 enough, this is the case, but sometimes it is not, and if it is not
156 then unexpected modification of the data or metadata can introduce
157 bugs that are very hard to debug and track down. You can check if a
158 buffer is writable with <function>gst_buffer_is_writable ()</function>.
159 Since you can pass back a different buffer than the one passed in,
160 it is a good idea to make the buffer writable in the callback function
161 with <function>gst_buffer_make_writable ()</function>.
164 Pad probes are suited best for looking at data as it passes through
165 the pipeline. If you need to modify data, you should better write your
166 own GStreamer element. Base classes like GstAudioFilter, GstVideoFilter or
167 GstBaseTransform make this fairly easy.
170 If you just want to inspect buffers as they pass through the pipeline,
171 you don't even need to set up pad probes. You could also just insert
172 an identity element into the pipeline and connect to its "handoff"
173 signal. The identity element also provides a few useful debugging tools
174 like the "dump" property or the "last-message" property (the latter is
175 enabled by passing the '-v' switch to gst-launch and by setting the
176 silent property on the identity to FALSE).
180 <sect1 id="section-data-spoof">
181 <title>Manually adding or removing data from/to a pipeline</title>
183 Many people have expressed the wish to use their own sources to inject
184 data into a pipeline. Some people have also expressed the wish to grab
185 the output in a pipeline and take care of the actual output inside
186 their application. While either of these methods are strongly
187 discouraged, &GStreamer; offers support for this.
188 <emphasis>Beware! You need to know what you are doing.</emphasis> Since
189 you don't have any support from a base class you need to thoroughly
190 understand state changes and synchronization. If it doesn't work,
191 there are a million ways to shoot yourself in the foot. It's always
192 better to simply write a plugin and have the base class manage it.
193 See the Plugin Writer's Guide for more information on this topic. Also
194 see the next section, which will explain how to embed plugins statically
198 There's two possible elements that you can use for the above-mentioned
199 purposes. Those are called <quote>appsrc</quote> (an imaginary source)
200 and <quote>appsink</quote> (an imaginary sink). The same method applies
201 to each of those elements. Here, we will discuss how to use those
202 elements to insert (using appsrc) or grab (using appsink) data from a
203 pipeline, and how to set negotiation.
206 Both appsrc and appsink provide 2 sets of API. One API uses standard
207 GObject (action) signals and properties. The same API is also
208 available as a regular C api. The C api is more performant but
209 requires you to link to the app library in order to use the elements.
212 <sect2 id="section-spoof-appsrc">
213 <title>Inserting data with appsrc</title>
215 First we look at some examples for appsrc, which lets you insert data
216 into the pipeline from the application. Appsrc has some configuration
217 options that define how it will operate. You should decide about the
218 following configurations:
223 Will the appsrc operate in push or pull mode. The stream-type
224 property can be used to control this. stream-type of
225 <quote>random-access</quote> will activate pull mode scheduling
226 while the other stream-types activate push mode.
231 The caps of the buffers that appsrc will push out. This needs to
232 be configured with the caps property. The caps must be set to a
233 fixed caps and will be used to negotiate a format downstream.
238 It the appsrc operates in live mode or not. This can be configured
239 with the is-live property. When operating in live-mode it is
240 important to configure the min-latency and max-latency in appsrc.
241 The min-latency should be set to the amount of time it takes between
242 capturing a buffer and when it is pushed inside appsrc.
243 In live mode, you should timestamp the buffers with the pipeline
244 running-time when the first byte of the buffer was captured before
245 feeding them to appsrc. You can let appsrc do the timestaping with
246 the do-timestamp property (but then the min-latency must be set
247 to 0 because it timestamps based on the running-time when the buffer
253 The format of the SEGMENT event that appsrc will push. The format
254 has implications for how the running-time of the buffers will
255 be calculated so you must be sure you understand this. For
256 live sources you probably want to set the format property to
257 GST_FORMAT_TIME. For non-live source it depends on the media type
258 that you are handling. If you plan to timestamp the buffers, you
259 should probably put a GST_FORMAT_TIME format, otherwise
260 GST_FORMAT_BYTES might be appropriate.
265 If appsrc operates in random-access mode, it is important to configure
266 the size property of appsrc with the number of bytes in the stream.
267 This will allow downstream elements to know the size of the media and
268 alows them to seek to the end of the stream when needed.
273 The main way of handling data to appsrc is by using the function
274 <function>gst_app_src_push_buffer ()</function> or by emiting the
275 push-buffer action signal. This will put the buffer onto a queue from
276 which appsrc will read from in its streaming thread. It is important
277 to note that data transport will not happen from the thread that
278 performed the push-buffer call.
281 The <quote>max-bytes</quote> property controls how much data can be
282 queued in appsrc before appsrc considers the queue full. A filled
283 internal queue will always signal the <quote>enough-data</quote>
284 signal, which signals the application that it should stop pushing
285 data into appsrc. The <quote>block</quote> property will cause appsrc to
286 block the push-buffer method until free data becomes available again.
289 When the internal queue is running out of data, the
290 <quote>need-data</quote> signal is emitted, which signals the application
291 that it should start pushing more data into appsrc.
294 In addition to the <quote>need-data</quote> and <quote>enough-data</quote>
295 signals, appsrc can emit the <quote>seek-data</quote> signal when the
296 <quote>stream-mode</quote> property is set to <quote>seekable</quote>
297 or <quote>random-access</quote>. The signal argument will contain the
298 new desired position in the stream expressed in the unit set with the
299 <quote>format</quote> property. After receiving the seek-data signal,
300 the application should push-buffers from the new position.
303 When the last byte is pushed into appsrc, you must call
304 <function>gst_app_src_end_of_stream ()</function> to make it send
308 These signals allow the application to operate appsrc in push and
309 pull mode as will be explained next.
312 <sect3 id="section-spoof-appsrc-push">
313 <title>Using appsrc in push mode</title>
315 When appsrc is configured in push mode (stream-type is stream or
316 seekable), the application repeatedly calls the push-buffer method
317 with a new buffer. Optionally, the queue size in the appsrc can be
318 controlled with the enough-data and need-data signals by respectively
319 stopping/starting the push-buffer calls. The value of the
320 min-percent property defines how empty the internal appsrc queue
321 needs to be before the need-data signal will be fired. You can set
322 this to some value >0 to avoid completely draining the queue.
325 When the stream-type is set to seekable, don't forget to implement
326 a seek-data callback.
329 Use this model when implementing various network protocols or
334 <sect3 id="section-spoof-appsrc-pull">
335 <title>Using appsrc in pull mode</title>
337 In the pull model, data is fed to appsrc from the need-data signal
338 handler. You should push exactly the amount of bytes requested in the
339 need-data signal. You are only allowed to push less bytes when you are
340 at the end of the stream.
343 Use this model for file access or other randomly accessable sources.
347 <sect3 id="section-spoof-appsrc-ex">
348 <title>Appsrc example</title>
350 This example application will generate black/white (it switches
351 every second) video to an Xv-window output by using appsrc as a
352 source with caps to force a format. We use a colorspace
353 conversion element to make sure that we feed the right format to
354 your X server. We configure a video stream with a variable framerate
355 (0/1) and we set the timestamps on the outgoing buffers in such
356 a way that we play 2 frames per second.
359 Note how we use the pull mode method of pushing new buffers into
360 appsrc although appsrc is running in push mode.
362 <programlisting><!-- example-begin appsrc.c -->
363 #include <gst/gst.h>
365 static GMainLoop *loop;
368 cb_need_data (GstElement *appsrc,
372 static gboolean white = FALSE;
373 static GstClockTime timestamp = 0;
378 size = 385 * 288 * 2;
380 buffer = gst_buffer_new_allocate (NULL, size, NULL);
382 /* this makes the image black/white */
383 gst_buffer_memset (buffer, 0, white ? 0xff : 0x0, size);
387 GST_BUFFER_PTS (buffer) = timestamp;
388 GST_BUFFER_DURATION (buffer) = gst_util_uint64_scale_int (1, GST_SECOND, 2);
390 timestamp += GST_BUFFER_DURATION (buffer);
392 g_signal_emit_by_name (appsrc, "push-buffer", buffer, &ret);
394 if (ret != GST_FLOW_OK) {
395 /* something wrong, stop pushing */
396 g_main_loop_quit (loop);
404 GstElement *pipeline, *appsrc, *conv, *videosink;
407 gst_init (&argc, &argv);
408 loop = g_main_loop_new (NULL, FALSE);
411 pipeline = gst_pipeline_new ("pipeline");
412 appsrc = gst_element_factory_make ("appsrc", "source");
413 conv = gst_element_factory_make ("videoconvert", "conv");
414 videosink = gst_element_factory_make ("xvimagesink", "videosink");
417 g_object_set (G_OBJECT (appsrc), "caps",
418 gst_caps_new_simple ("video/x-raw",
419 "format", G_TYPE_STRING, "RGB16",
420 "width", G_TYPE_INT, 384,
421 "height", G_TYPE_INT, 288,
422 "framerate", GST_TYPE_FRACTION, 0, 1,
424 gst_bin_add_many (GST_BIN (pipeline), appsrc, conv, videosink, NULL);
425 gst_element_link_many (appsrc, conv, videosink, NULL);
428 g_object_set (G_OBJECT (appsrc),
430 "format", GST_FORMAT_TIME, NULL);
431 g_signal_connect (appsrc, "need-data", G_CALLBACK (cb_need_data), NULL);
434 gst_element_set_state (pipeline, GST_STATE_PLAYING);
435 g_main_loop_run (loop);
438 gst_element_set_state (pipeline, GST_STATE_NULL);
439 gst_object_unref (GST_OBJECT (pipeline));
440 g_main_loop_unref (loop);
444 <!-- example-end appsrc.c --></programlisting>
448 <sect2 id="section-spoof-appsink">
449 <title>Grabbing data with appsink</title>
451 Unlike appsrc, appsink is a little easier to use. It also supports
452 a pull and push based model of getting data from the pipeline.
455 The normal way of retrieving samples from appsink is by using the
456 <function>gst_app_sink_pull_sample()</function> and
457 <function>gst_app_sink_pull_preroll()</function> methods or by using
458 the <quote>pull-sample</quote> and <quote>pull-preroll</quote>
459 signals. These methods block until a sample becomes available in the
460 sink or when the sink is shut down or reaches EOS.
463 Appsink will internally use a queue to collect buffers from the
464 streaming thread. If the application is not pulling samples fast
465 enough, this queue will consume a lot of memory over time. The
466 <quote>max-buffers</quote> property can be used to limit the queue
467 size. The <quote>drop</quote> property controls whether the
468 streaming thread blocks or if older buffers are dropped when the
469 maximum queue size is reached. Note that blocking the streaming thread
470 can negatively affect real-time performance and should be avoided.
473 If a blocking behaviour is not desirable, setting the
474 <quote>emit-signals</quote> property to TRUE will make appsink emit
475 the <quote>new-sample</quote> and <quote>new-preroll</quote> signals
476 when a sample can be pulled without blocking.
479 The <quote>caps</quote> property on appsink can be used to control
480 the formats that appsink can receive. This property can contain
481 non-fixed caps, the format of the pulled samples can be obtained by
482 getting the sample caps.
485 If one of the pull-preroll or pull-sample methods return NULL, the
486 appsink is stopped or in the EOS state. You can check for the EOS state
487 with the <quote>eos</quote> property or with the
488 <function>gst_app_sink_is_eos()</function> method.
491 The eos signal can also be used to be informed when the EOS state is
492 reached to avoid polling.
495 Consider configuring the following properties in the appsink:
500 The <quote>sync</quote> property if you want to have the sink
501 base class synchronize the buffer against the pipeline clock
502 before handing you the sample.
507 Enable Quality-of-Service with the <quote>qos</quote> property.
508 If you are dealing with raw video frames and let the base class
509 sycnhronize on the clock, it might be a good idea to also let
510 the base class send QOS events upstream.
515 The caps property that contains the accepted caps. Upstream elements
516 will try to convert the format so that it matches the configured
517 caps on appsink. You must still check the
518 <classname>GstSample</classname> to get the actual caps of the
524 <sect3 id="section-spoof-appsink-ex">
525 <title>Appsink example</title>
527 What follows is an example on how to capture a snapshot of a video
528 stream using appsink.
532 <!-- example-begin appsink.c -->
540 #define CAPS "video/x-raw,format=RGB,width=160,pixel-aspect-ratio=1/1"
543 main (int argc, char *argv[])
545 GstElement *pipeline, *sink;
549 GError *error = NULL;
550 gint64 duration, position;
551 GstStateChangeReturn ret;
555 gst_init (&argc, &argv);
558 g_print ("usage: %s <uri>\n Writes snapshot.png in the current directory\n",
563 /* create a new pipeline */
565 g_strdup_printf ("uridecodebin uri=%s ! videoconvert ! videoscale ! "
566 " appsink name=sink caps=\"" CAPS "\"", argv[1]);
567 pipeline = gst_parse_launch (descr, &error);
570 g_print ("could not construct pipeline: %s\n", error->message);
571 g_error_free (error);
576 sink = gst_bin_get_by_name (GST_BIN (pipeline), "sink");
578 /* set to PAUSED to make the first frame arrive in the sink */
579 ret = gst_element_set_state (pipeline, GST_STATE_PAUSED);
581 case GST_STATE_CHANGE_FAILURE:
582 g_print ("failed to play the file\n");
584 case GST_STATE_CHANGE_NO_PREROLL:
585 /* for live sources, we need to set the pipeline to PLAYING before we can
586 * receive a buffer. We don't do that yet */
587 g_print ("live sources not supported yet\n");
592 /* This can block for up to 5 seconds. If your machine is really overloaded,
593 * it might time out before the pipeline prerolled and we generate an error. A
594 * better way is to run a mainloop and catch errors there. */
595 ret = gst_element_get_state (pipeline, NULL, NULL, 5 * GST_SECOND);
596 if (ret == GST_STATE_CHANGE_FAILURE) {
597 g_print ("failed to play the file\n");
601 /* get the duration */
602 gst_element_query_duration (pipeline, GST_FORMAT_TIME, &duration);
605 /* we have a duration, seek to 5% */
606 position = duration * 5 / 100;
608 /* no duration, seek to 1 second, this could EOS */
609 position = 1 * GST_SECOND;
611 /* seek to the a position in the file. Most files have a black first frame so
612 * by seeking to somewhere else we have a bigger chance of getting something
613 * more interesting. An optimisation would be to detect black images and then
614 * seek a little more */
615 gst_element_seek_simple (pipeline, GST_FORMAT_TIME,
616 GST_SEEK_FLAG_KEY_UNIT | GST_SEEK_FLAG_FLUSH, position);
618 /* get the preroll buffer from appsink, this block untils appsink really
620 g_signal_emit_by_name (sink, "pull-preroll", &sample, NULL);
622 /* if we have a buffer now, convert it to a pixbuf. It's possible that we
623 * don't have a buffer because we went EOS right away or had an error. */
629 /* get the snapshot buffer format now. We set the caps on the appsink so
630 * that it can only be an rgb buffer. The only thing we have not specified
631 * on the caps is the height, which is dependant on the pixel-aspect-ratio
632 * of the source material */
633 caps = gst_sample_get_caps (sample);
635 g_print ("could not get snapshot format\n");
638 s = gst_caps_get_structure (caps, 0);
640 /* we need to get the final caps on the buffer to get the size */
641 res = gst_structure_get_int (s, "width", &width);
642 res |= gst_structure_get_int (s, "height", &height);
644 g_print ("could not get snapshot dimension\n");
648 /* create pixmap from buffer and save, gstreamer video buffers have a stride
649 * that is rounded up to the nearest multiple of 4 */
650 buffer = gst_sample_get_buffer (sample);
651 gst_buffer_map (buffer, &map, GST_MAP_READ);
653 pixbuf = gdk_pixbuf_new_from_data (map.data,
654 GDK_COLORSPACE_RGB, FALSE, 8, width, height,
655 GST_ROUND_UP_4 (width * 3), NULL, NULL);
657 /* save the pixbuf */
658 gdk_pixbuf_save (pixbuf, "snapshot.png", "png", &error, NULL);
660 gst_buffer_unmap (buffer, &map);
662 g_print ("could not make snapshot\n");
665 /* cleanup and exit */
666 gst_element_set_state (pipeline, GST_STATE_NULL);
667 gst_object_unref (pipeline);
672 <!-- example-end appsink.c --></programlisting>
677 <sect1 id="section-spoof-format">
678 <title>Forcing a format</title>
680 Sometimes you'll want to set a specific format, for example a video
681 size and format or an audio bitsize and number of channels. You can
682 do this by forcing a specific <classname>GstCaps</classname> on
683 the pipeline, which is possible by using
684 <emphasis>filtered caps</emphasis>. You can set a filtered caps on
685 a link by using the <quote>capsfilter</quote> element in between the
686 two elements, and specifying a <classname>GstCaps</classname> as
687 <quote>caps</quote> property on this element. It will then
688 only allow types matching that specified capability set for
689 negotiation. See also <xref linkend="section-caps-filter"/>.
693 <sect1 id="section-spoof-probes">
694 <title>Dynamically changing the pipeline</title>
700 <sect1 id="section-data-manager">
701 <title>Embedding static elements in your application</title>
703 The <ulink type="http"
704 url="http://gstreamer.freedesktop.org/data/doc/gstreamer/head/pwg/html/index.html">Plugin
705 Writer's Guide</ulink> describes in great detail how to write elements
706 for the &GStreamer; framework. In this section, we will solely discuss
707 how to embed such elements statically in your application. This can be
708 useful for application-specific elements that have no use elsewhere in
712 Dynamically loaded plugins contain a structure that's defined using
713 <function>GST_PLUGIN_DEFINE ()</function>. This structure is loaded
714 when the plugin is loaded by the &GStreamer; core. The structure
715 contains an initialization function (usually called
716 <function>plugin_init</function>) that will be called right after that.
717 It's purpose is to register the elements provided by the plugin with
718 the &GStreamer; framework.
719 If you want to embed elements directly in
720 your application, the only thing you need to do is to replace
721 <function>GST_PLUGIN_DEFINE ()</function> with a call to
722 <function>gst_plugin_register_static ()</function>. As soon as you
723 call <function>gst_plugin_register_static ()</function>, the elements
724 will from then on be available like any other element, without them
725 having to be dynamically loadable libraries. In the example below, you
726 would be able to call <function>gst_element_factory_make
727 ("my-element-name", "some-name")</function> to create an instance of the
733 * Here, you would write the actual plugin code.
739 register_elements (GstPlugin *plugin)
741 return gst_element_register (plugin, "my-element-name",
742 GST_RANK_NONE, MY_PLUGIN_TYPE);
750 gst_plugin_register_static (
753 "my-private-plugins",
754 "Private elements of my application",
758 "my-application-source",
760 "http://www.my-application.net/")