1 # Basic tutorial 8: Short-cutting the pipeline
5 Pipelines constructed with GStreamer do not need to be completely
6 closed. Data can be injected into the pipeline and extracted from it at
7 any time, in a variety of ways. This tutorial shows:
9 - How to inject external data into a general GStreamer pipeline.
11 - How to extract data from a general GStreamer pipeline.
13 - How to access and manipulate this data.
15 [](tutorials/playback/short-cutting-the-pipeline.md) explains
16 how to achieve the same goals in a playbin-based pipeline.
20 Applications can interact with the data flowing through a GStreamer
21 pipeline in several ways. This tutorial describes the easiest one, since
22 it uses elements that have been created for this sole purpose.
24 The element used to inject application data into a GStreamer pipeline is
25 `appsrc`, and its counterpart, used to extract GStreamer data back to
26 the application is `appsink`. To avoid confusing the names, think of it
27 from GStreamer's point of view: `appsrc` is just a regular source, that
28 provides data magically fallen from the sky (provided by the
29 application, actually). `appsink` is a regular sink, where the data
30 flowing through a GStreamer pipeline goes to die (it is recovered by the
31 application, actually).
33 `appsrc` and `appsink` are so versatile that they offer their own API
34 (see their documentation), which can be accessed by linking against the
35 `gstreamer-app` library. In this tutorial, however, we will use a
36 simpler approach and control them through signals.
38 `appsrc` can work in a variety of modes: in **pull** mode, it requests
39 data from the application every time it needs it. In **push** mode, the
40 application pushes data at its own pace. Furthermore, in push mode, the
41 application can choose to be blocked in the push function when enough
42 data has already been provided, or it can listen to the
43 `enough-data` and `need-data` signals to control flow. This example
44 implements the latter approach. Information regarding the other methods
45 can be found in the `appsrc` documentation.
49 Data travels through a GStreamer pipeline in chunks called **buffers**.
50 Since this example produces and consumes data, we need to know about
53 Source Pads produce buffers, that are consumed by Sink Pads; GStreamer
54 takes these buffers and passes them from element to element.
56 A buffer simply represents a unit of data, do not assume that all
57 buffers will have the same size, or represent the same amount of time.
58 Neither should you assume that if a single buffer enters an element, a
59 single buffer will come out. Elements are free to do with the received
60 buffers as they please. `GstBuffer`s may also contain more than one
61 actual memory buffer. Actual memory buffers are abstracted away using
62 `GstMemory` objects, and a `GstBuffer` can contain multiple `GstMemory` objects.
64 Every buffer has attached time-stamps and duration, that describe in
65 which moment the content of the buffer should be decoded, rendered or
66 displayed. Time stamping is a very complex and delicate subject, but
67 this simplified vision should suffice for now.
69 As an example, a `filesrc` (a GStreamer element that reads files)
70 produces buffers with the “ANY” caps and no time-stamping information.
71 After demuxing (see [](tutorials/basic/dynamic-pipelines.md))
72 buffers can have some specific caps, for example “video/x-h264”. After
73 decoding, each buffer will contain a single video frame with raw caps
74 (for example, “video/x-raw-yuv”) and very precise time stamps indicating
75 when should that frame be displayed.
79 This tutorial expands [](tutorials/basic/multithreading-and-pad-availability.md) in
80 two ways: firstly, the `audiotestsrc` is replaced by an `appsrc` that
81 will generate the audio data. Secondly, a new branch is added to the
82 `tee` so data going into the audio sink and the wave display is also
83 replicated into an `appsink`. The `appsink` uploads the information back
84 into the application, which then just notifies the user that data has
85 been received, but it could obviously perform more complex tasks.
87 ![](images/tutorials/basic-tutorial-8.png)
89 ## A crude waveform generator
91 Copy this code into a text file named `basic-tutorial-8.c` (or find it
92 in the SDK installation).
96 #include <gst/audio/audio.h>
99 #define CHUNK_SIZE 1024 /* Amount of bytes we are sending in each buffer */
100 #define SAMPLE_RATE 44100 /* Samples per second we are sending */
102 /* Structure to contain all our information, so we can pass it to callbacks */
103 typedef struct _CustomData {
104 GstElement *pipeline, *app_source, *tee, *audio_queue, *audio_convert1, *audio_resample, *audio_sink;
105 GstElement *video_queue, *audio_convert2, *visual, *video_convert, *video_sink;
106 GstElement *app_queue, *app_sink;
108 guint64 num_samples; /* Number of samples generated so far (for timestamp generation) */
109 gfloat a, b, c, d; /* For waveform generation */
111 guint sourceid; /* To control the GSource */
113 GMainLoop *main_loop; /* GLib's Main Loop */
116 /* This method is called by the idle GSource in the mainloop, to feed CHUNK_SIZE bytes into appsrc.
117 * The ide handler is added to the mainloop when appsrc requests us to start sending data (need-data signal)
118 * and is removed when appsrc has enough data (enough-data signal).
120 static gboolean push_data (CustomData *data) {
126 gint num_samples = CHUNK_SIZE / 2; /* Because each sample is 16 bits */
129 /* Create a new empty buffer */
130 buffer = gst_buffer_new_and_alloc (CHUNK_SIZE);
132 /* Set its timestamp and duration */
133 GST_BUFFER_TIMESTAMP (buffer) = gst_util_uint64_scale (data->num_samples, GST_SECOND, SAMPLE_RATE);
134 GST_BUFFER_DURATION (buffer) = gst_util_uint64_scale (CHUNK_SIZE, GST_SECOND, SAMPLE_RATE);
136 /* Generate some psychodelic waveforms */
137 gst_buffer_map (buffer, &map, GST_MAP_WRITE);
138 raw = (gint16 *)map.data;
140 data->d -= data->c / 1000;
141 freq = 1100 + 1000 * data->d;
142 for (i = 0; i < num_samples; i++) {
144 data->b -= data->a / freq;
145 raw[i] = (gint16)(500 * data->a);
147 gst_buffer_unmap (buffer, &map);
148 data->num_samples += num_samples;
150 /* Push the buffer into the appsrc */
151 g_signal_emit_by_name (data->app_source, "push-buffer", buffer, &ret);
153 /* Free the buffer now that we are done with it */
154 gst_buffer_unref (buffer);
156 if (ret != GST_FLOW_OK) {
157 /* We got some error, stop sending data */
164 /* This signal callback triggers when appsrc needs data. Here, we add an idle handler
165 * to the mainloop to start pushing data into the appsrc */
166 static void start_feed (GstElement *source, guint size, CustomData *data) {
167 if (data->sourceid == 0) {
168 g_print ("Start feeding\n");
169 data->sourceid = g_idle_add ((GSourceFunc) push_data, data);
173 /* This callback triggers when appsrc has enough data and we can stop sending.
174 * We remove the idle handler from the mainloop */
175 static void stop_feed (GstElement *source, CustomData *data) {
176 if (data->sourceid != 0) {
177 g_print ("Stop feeding\n");
178 g_source_remove (data->sourceid);
183 /* The appsink has received a buffer */
184 static void new_sample (GstElement *sink, CustomData *data) {
187 /* Retrieve the buffer */
188 g_signal_emit_by_name (sink, "pull-sample", &sample);
190 /* The only thing we do in this example is print a * to indicate a received buffer */
192 gst_buffer_unref (sample);
196 /* This function is called when an error message is posted on the bus */
197 static void error_cb (GstBus *bus, GstMessage *msg, CustomData *data) {
201 /* Print error details on the screen */
202 gst_message_parse_error (msg, &err, &debug_info);
203 g_printerr ("Error received from element %s: %s\n", GST_OBJECT_NAME (msg->src), err->message);
204 g_printerr ("Debugging information: %s\n", debug_info ? debug_info : "none");
205 g_clear_error (&err);
208 g_main_loop_quit (data->main_loop);
211 int main(int argc, char *argv[]) {
213 GstPadTemplate *tee_src_pad_template;
214 GstPad *tee_audio_pad, *tee_video_pad, *tee_app_pad;
215 GstPad *queue_audio_pad, *queue_video_pad, *queue_app_pad;
220 /* Initialize cumstom data structure */
221 memset (&data, 0, sizeof (data));
222 data.b = 1; /* For waveform generation */
225 /* Initialize GStreamer */
226 gst_init (&argc, &argv);
228 /* Create the elements */
229 data.app_source = gst_element_factory_make ("appsrc", "audio_source");
230 data.tee = gst_element_factory_make ("tee", "tee");
231 data.audio_queue = gst_element_factory_make ("queue", "audio_queue");
232 data.audio_convert1 = gst_element_factory_make ("audioconvert", "audio_convert1");
233 data.audio_resample = gst_element_factory_make ("audioresample", "audio_resample");
234 data.audio_sink = gst_element_factory_make ("autoaudiosink", "audio_sink");
235 data.video_queue = gst_element_factory_make ("queue", "video_queue");
236 data.audio_convert2 = gst_element_factory_make ("audioconvert", "audio_convert2");
237 data.visual = gst_element_factory_make ("wavescope", "visual");
238 data.video_convert = gst_element_factory_make ("videoconvert", "csp");
239 data.video_sink = gst_element_factory_make ("autovideosink", "video_sink");
240 data.app_queue = gst_element_factory_make ("queue", "app_queue");
241 data.app_sink = gst_element_factory_make ("appsink", "app_sink");
243 /* Create the empty pipeline */
244 data.pipeline = gst_pipeline_new ("test-pipeline");
246 if (!data.pipeline || !data.app_source || !data.tee || !data.audio_queue || !data.audio_convert1 ||
247 !data.audio_resample || !data.audio_sink || !data.video_queue || !data.audio_convert2 || !data.visual ||
248 !data.video_convert || !data.video_sink || !data.app_queue || !data.app_sink) {
249 g_printerr ("Not all elements could be created.\n");
253 /* Configure wavescope */
254 g_object_set (data.visual, "shader", 0, "style", 0, NULL);
256 /* Configure appsrc */
257 gst_audio_info_set_format (&info, GST_AUDIO_FORMAT_S16, SAMPLE_RATE, 1, NULL);
258 audio_caps = gst_audio_info_to_caps (&info);
259 g_object_set (data.app_source, "caps", audio_caps, "format", GST_FORMAT_TIME, NULL);
260 g_signal_connect (data.app_source, "need-data", G_CALLBACK (start_feed), &data);
261 g_signal_connect (data.app_source, "enough-data", G_CALLBACK (stop_feed), &data);
263 /* Configure appsink */
264 g_object_set (data.app_sink, "emit-signals", TRUE, "caps", audio_caps, NULL);
265 g_signal_connect (data.app_sink, "new-sample", G_CALLBACK (new_sample), &data);
266 gst_caps_unref (audio_caps);
267 g_free (audio_caps_text);
269 /* Link all elements that can be automatically linked because they have "Always" pads */
270 gst_bin_add_many (GST_BIN (data.pipeline), data.app_source, data.tee, data.audio_queue, data.audio_convert1, data.audio_resample,
271 data.audio_sink, data.video_queue, data.audio_convert2, data.visual, data.video_convert, data.video_sink, data.app_queue,
272 data.app_sink, NULL);
273 if (gst_element_link_many (data.app_source, data.tee, NULL) != TRUE ||
274 gst_element_link_many (data.audio_queue, data.audio_convert1, data.audio_resample, data.audio_sink, NULL) != TRUE ||
275 gst_element_link_many (data.video_queue, data.audio_convert2, data.visual, data.video_convert, data.video_sink, NULL) != TRUE ||
276 gst_element_link_many (data.app_queue, data.app_sink, NULL) != TRUE) {
277 g_printerr ("Elements could not be linked.\n");
278 gst_object_unref (data.pipeline);
282 /* Manually link the Tee, which has "Request" pads */
283 tee_src_pad_template = gst_element_class_get_pad_template (GST_ELEMENT_GET_CLASS (data.tee), "src_%d");
284 tee_audio_pad = gst_element_request_pad (data.tee, tee_src_pad_template, NULL, NULL);
285 g_print ("Obtained request pad %s for audio branch.\n", gst_pad_get_name (tee_audio_pad));
286 queue_audio_pad = gst_element_get_static_pad (data.audio_queue, "sink");
287 tee_video_pad = gst_element_request_pad (data.tee, tee_src_pad_template, NULL, NULL);
288 g_print ("Obtained request pad %s for video branch.\n", gst_pad_get_name (tee_video_pad));
289 queue_video_pad = gst_element_get_static_pad (data.video_queue, "sink");
290 tee_app_pad = gst_element_request_pad (data.tee, tee_src_pad_template, NULL, NULL);
291 g_print ("Obtained request pad %s for app branch.\n", gst_pad_get_name (tee_app_pad));
292 queue_app_pad = gst_element_get_static_pad (data.app_queue, "sink");
293 if (gst_pad_link (tee_audio_pad, queue_audio_pad) != GST_PAD_LINK_OK ||
294 gst_pad_link (tee_video_pad, queue_video_pad) != GST_PAD_LINK_OK ||
295 gst_pad_link (tee_app_pad, queue_app_pad) != GST_PAD_LINK_OK) {
296 g_printerr ("Tee could not be linked\n");
297 gst_object_unref (data.pipeline);
300 gst_object_unref (queue_audio_pad);
301 gst_object_unref (queue_video_pad);
302 gst_object_unref (queue_app_pad);
304 /* Instruct the bus to emit signals for each received message, and connect to the interesting signals */
305 bus = gst_element_get_bus (data.pipeline);
306 gst_bus_add_signal_watch (bus);
307 g_signal_connect (G_OBJECT (bus), "message::error", (GCallback)error_cb, &data);
308 gst_object_unref (bus);
310 /* Start playing the pipeline */
311 gst_element_set_state (data.pipeline, GST_STATE_PLAYING);
313 /* Create a GLib Main Loop and set it to run */
314 data.main_loop = g_main_loop_new (NULL, FALSE);
315 g_main_loop_run (data.main_loop);
317 /* Release the request pads from the Tee, and unref them */
318 gst_element_release_request_pad (data.tee, tee_audio_pad);
319 gst_element_release_request_pad (data.tee, tee_video_pad);
320 gst_element_release_request_pad (data.tee, tee_app_pad);
321 gst_object_unref (tee_audio_pad);
322 gst_object_unref (tee_video_pad);
323 gst_object_unref (tee_app_pad);
326 gst_element_set_state (data.pipeline, GST_STATE_NULL);
327 gst_object_unref (data.pipeline);
332 > ![Information](images/icons/emoticons/information.png)
335 > If you need help to compile this code, refer to the **Building the tutorials** section for your platform: [Linux](installing/on-linux.md#InstallingonLinux-Build), [Mac OS X](installing/on-mac-osx.md#InstallingonMacOSX-Build) or [Windows](installing/on-windows.md#InstallingonWindows-Build), or use this specific command on Linux:
337 > `` gcc basic-tutorial-8.c -o basic-tutorial-8 `pkg-config --cflags --libs gstreamer-1.0 gst-audio-1.0` ``
339 >If you need help to run this code, refer to the **Running the tutorials** section for your platform: [Linux](installing/on-linux.md#InstallingonLinux-Run), [Mac OS X](installing/on-mac-osx.md#InstallingonMacOSX-Run) or [Windows](installing/on-windows.md#InstallingonWindows-Run).
341 > This tutorial plays an audible tone for varying frequency through the audio card and opens a window with a waveform representation of the tone. The waveform should be a sinusoid, but due to the refreshing of the window might not appear so.
343 > Required libraries: `gstreamer-1.0`
347 The code to create the pipeline (Lines 131 to 205) is an enlarged
348 version of [Basic tutorial 7: Multithreading and Pad
349 Availability](tutorials/basic/multithreading-and-pad-availability.md).
350 It involves instantiating all the elements, link the elements with
351 Always Pads, and manually link the Request Pads of the `tee` element.
353 Regarding the configuration of the `appsrc` and `appsink` elements:
356 /* Configure appsrc */
357 audio_caps_text = g_strdup_printf (AUDIO_CAPS, SAMPLE_RATE);
358 audio_caps = gst_caps_from_string (audio_caps_text);
359 g_object_set (data.app_source, "caps", audio_caps, NULL);
360 g_signal_connect (data.app_source, "need-data", G_CALLBACK (start_feed), &data);
361 g_signal_connect (data.app_source, "enough-data", G_CALLBACK (stop_feed), &data);
364 The first property that needs to be set on the `appsrc` is `caps`. It
365 specifies the kind of data that the element is going to produce, so
366 GStreamer can check if linking with downstream elements is possible
367 (this is, if the downstream elements will understand this kind of data).
368 This property must be a `GstCaps` object, which is easily built from a
369 string with `gst_caps_from_string()`.
371 We then connect to the `need-data` and `enough-data` signals. These are
372 fired by `appsrc` when its internal queue of data is running low or
373 almost full, respectively. We will use these signals to start and stop
374 (respectively) our signal generation process.
377 /* Configure appsink */
378 g_object_set (data.app_sink, "emit-signals", TRUE, "caps", audio_caps, NULL);
379 g_signal_connect (data.app_sink, "new-sample", G_CALLBACK (new_sample), &data);
380 gst_caps_unref (audio_caps);
381 g_free (audio_caps_text);
384 Regarding the `appsink` configuration, we connect to the
385 `new-sample` signal, which is emitted every time the sink receives a
386 buffer. Also, the signal emission needs to be enabled through the
387 `emit-signals` property, because, by default, it is disabled.
389 Starting the pipeline, waiting for messages and final cleanup is done as
390 usual. Let's review the callbacks we have just
394 /* This signal callback triggers when appsrc needs data. Here, we add an idle handler
395 * to the mainloop to start pushing data into the appsrc */
396 static void start_feed (GstElement *source, guint size, CustomData *data) {
397 if (data->sourceid == 0) {
398 g_print ("Start feeding\n");
399 data->sourceid = g_idle_add ((GSourceFunc) push_data, data);
404 This function is called when the internal queue of `appsrc` is about to
405 starve (run out of data). The only thing we do here is register a GLib
406 idle function with `g_idle_add()` that feeds data to `appsrc` until it
407 is full again. A GLib idle function is a method that GLib will call from
408 its main loop whenever it is “idle”, this is, when it has no
409 higher-priority tasks to perform. It requires a GLib `GMainLoop` to be
410 instantiated and running, obviously.
412 This is only one of the multiple approaches that `appsrc` allows. In
413 particular, buffers do not need to be fed into `appsrc` from the main
414 thread using GLib, and you do not need to use the `need-data` and
415 `enough-data` signals to synchronize with `appsrc` (although this is
416 allegedly the most convenient).
418 We take note of the sourceid that `g_idle_add()` returns, so we can
423 /* This callback triggers when appsrc has enough data and we can stop sending.
424 * We remove the idle handler from the mainloop */
425 static void stop_feed (GstElement *source, CustomData *data) {
426 if (data->sourceid != 0) {
427 g_print ("Stop feeding\n");
428 g_source_remove (data->sourceid);
434 This function is called when the internal queue of `appsrc` is full
435 enough so we stop pushing data. Here we simply remove the idle function
436 by using `g_source_remove()` (The idle function is implemented as a
440 /* This method is called by the idle GSource in the mainloop, to feed CHUNK_SIZE bytes into appsrc.
441 * The ide handler is added to the mainloop when appsrc requests us to start sending data (need-data signal)
442 * and is removed when appsrc has enough data (enough-data signal).
444 static gboolean push_data (CustomData *data) {
449 gint num_samples = CHUNK_SIZE / 2; /* Because each sample is 16 bits */
452 /* Create a new empty buffer */
453 buffer = gst_buffer_new_and_alloc (CHUNK_SIZE);
455 /* Set its timestamp and duration */
456 GST_BUFFER_TIMESTAMP (buffer) = gst_util_uint64_scale (data->num_samples, GST_SECOND, SAMPLE_RATE);
457 GST_BUFFER_DURATION (buffer) = gst_util_uint64_scale (CHUNK_SIZE, GST_SECOND, SAMPLE_RATE);
459 /* Generate some psychodelic waveforms */
460 raw = (gint16 *)GST_BUFFER_DATA (buffer);
463 This is the function that feeds `appsrc`. It will be called by GLib at
464 times and rates which are out of our control, but we know that we will
465 disable it when its job is done (when the queue in `appsrc` is full).
467 Its first task is to create a new buffer with a given size (in this
468 example, it is arbitrarily set to 1024 bytes) with
469 `gst_buffer_new_and_alloc()`.
471 We count the number of samples that we have generated so far with the
472 `CustomData.num_samples` variable, so we can time-stamp this buffer
473 using the `GST_BUFFER_TIMESTAMP` macro in `GstBuffer`.
475 Since we are producing buffers of the same size, their duration is the
476 same and is set using the `GST_BUFFER_DURATION` in `GstBuffer`.
478 `gst_util_uint64_scale()` is a utility function that scales (multiply
479 and divide) numbers which can be large, without fear of overflows.
481 The bytes that for the buffer can be accessed with GST\_BUFFER\_DATA in
482 `GstBuffer` (Be careful not to write past the end of the buffer: you
483 allocated it, so you know its size).
485 We will skip over the waveform generation, since it is outside the scope
486 of this tutorial (it is simply a funny way of generating a pretty
490 /* Push the buffer into the appsrc */
491 g_signal_emit_by_name (data->app_source, "push-buffer", buffer, &ret);
493 /* Free the buffer now that we are done with it */
494 gst_buffer_unref (buffer);
497 Once we have the buffer ready, we pass it to `appsrc` with the
498 `push-buffer` action signal (see information box at the end of [](tutorials/playback/playbin-usage.md)), and then
499 `gst_buffer_unref()` it since we no longer need it.
502 /* The appsink has received a buffer */
503 static void new_sample (GstElement *sink, CustomData *data) {
505 /* Retrieve the buffer */
506 g_signal_emit_by_name (sink, "pull-sample", &sample);
508 /* The only thing we do in this example is print a * to indicate a received buffer */
510 gst_sample_unref (sample);
515 Finally, this is the function that gets called when the
516 `appsink` receives a buffer. We use the `pull-sample` action signal to
517 retrieve the buffer and then just print some indicator on the screen. We
518 can retrieve the data pointer using the `GST_BUFFER_DATA` macro and the
519 data size using the `GST_BUFFER_SIZE` macro in `GstBuffer`. Remember
520 that this buffer does not have to match the buffer that we produced in
521 the `push_data` function, any element in the path could have altered the
522 buffers in any way (Not in this example: there is only a `tee` in the
523 path between `appsrc` and `appsink`, and it does not change the content
526 We then `gst_buffer_unref()` the buffer, and this tutorial is done.
530 This tutorial has shown how applications can:
532 - Inject data into a pipeline using the `appsrc`element.
533 - Retrieve data from a pipeline using the `appsink` element.
534 - Manipulate this data by accessing the `GstBuffer`.
536 In a playbin-based pipeline, the same goals are achieved in a slightly
537 different way. [](tutorials/playback/short-cutting-the-pipeline.md) shows
540 It has been a pleasure having you here, and see you soon\!