3 This part gives an overview of the design of GStreamer with references
4 to the more detailed explanations of the different topics.
6 This document is intented for people that want to have a global overview
7 of the inner workings of GStreamer.
11 GStreamer is a set of libraries and plugins that can be used to
12 implement various multimedia applications ranging from desktop players,
13 audio/video recorders, multimedia servers, transcoders, etc.
15 Applications are built by constructing a pipeline composed of elements.
16 An element is an object that performs some action on a multimedia stream
20 - decode or encode between formats
21 - capture from a hardware device
22 - render to a hardware device
23 - mix or multiplex multiple streams
25 Elements have input and output pads called sink and source pads in
26 GStreamer. An application links elements together on pads to construct a
27 pipeline. Below is an example of an ogg/vorbis playback pipeline.
30 +-----------------------------------------------------------+
31 | ----------> downstream -------------------> |
34 | +---------+ +----------+ +-----------+ +----------+ |
35 | | filesrc | | oggdemux | | vorbisdec | | alsasink | |
36 | | src-sink src-sink src-sink | |
37 | +---------+ +----------+ +-----------+ +----------+ |
39 | <---------< upstream <-------------------< |
40 +-----------------------------------------------------------+
43 The filesrc element reads data from a file on disk. The oggdemux element
44 demultiplexes the data and sends a compressed audio stream to the vorbisdec
45 element. The vorbisdec element decodes the compressed data and sends it
46 to the alsasink element. The alsasink element sends the samples to the
47 audio card for playback.
49 Downstream and upstream are the terms used to describe the direction in
50 the Pipeline. From source to sink is called "downstream" and "upstream"
51 is from sink to source. Dataflow always happens downstream.
53 The task of the application is to construct a pipeline as above using
54 existing elements. This is further explained in the pipeline building
57 The application does not have to manage any of the complexities of the
58 actual dataflow/decoding/conversions/synchronisation etc. but only calls
59 high level functions on the pipeline object such as PLAY/PAUSE/STOP.
61 The application also receives messages and notifications from the
62 pipeline such as metadata, warning, error and EOS messages.
64 If the application needs more control over the graph it is possible to
65 directly access the elements and pads in the pipeline.
69 GStreamer design goals include:
71 - Process large amounts of data quickly
72 - Allow fully multithreaded processing
73 - Ability to deal with multiple formats
74 - Synchronize different dataflows
75 - Ability to deal with multiple devices
77 The capabilities presented to the application depends on the number of
78 elements installed on the system and their functionality.
80 The GStreamer core is designed to be media agnostic but provides many
81 features to elements to describe media formats.
85 The smallest building blocks in a pipeline are elements. An element
86 provides a number of pads which can be source or sinkpads. Sourcepads
87 provide data and sinkpads consume data. Below is an example of an ogg
88 demuxer element that has one pad that takes (sinks) data and two source
89 pads that produce data.
99 An element can be in four different states: `NULL`, `READY`, `PAUSED`,
100 `PLAYING`. In the `NULL` and `READY` state, the element is not processing any
101 data. In the `PLAYING` state it is processing data. The intermediate
102 PAUSED state is used to preroll data in the pipeline. A state change can
103 be performed with `gst_element_set_state()`.
105 An element always goes through all the intermediate state changes. This
106 means that when en element is in the `READY` state and is put to `PLAYING`,
107 it will first go through the intermediate `PAUSED` state.
109 An element state change to `PAUSED` will activate the pads of the element.
110 First the source pads are activated, then the sinkpads. When the pads
111 are activated, the pad activate function is called. Some pads will start
112 a thread (`GstTask`) or some other mechanism to start producing or
115 The `PAUSED` state is special as it is used to preroll data in the
116 pipeline. The purpose is to fill all connected elements in the pipeline
117 with data so that the subsequent `PLAYING` state change happens very
118 quickly. Some elements will therefore not complete the state change to
119 `PAUSED` before they have received enough data. Sink elements are required
120 to only complete the state change to `PAUSED` after receiving the first
123 Normally the state changes of elements are coordinated by the pipeline
124 as explained in [states](design/states.md).
126 Different categories of elements exist:
128 - *source elements*: these are elements that do not consume data but
129 only provide data for the pipeline.
131 - *sink elements*: these are elements that do not produce data but
132 renders data to an output device.
134 - *transform elements*: these elements transform an input stream in a
135 certain format into a stream of another format.
136 Encoder/decoder/converters are examples.
138 - *demuxer elements*: these elements parse a stream and produce several
141 - *mixer/muxer elements*: combine several input streams into one output
144 Other categories of elements can be constructed (see [klass](design/draft-klass.md)).
148 A bin is an element subclass and acts as a container for other elements
149 so that multiple elements can be combined into one element.
151 A bin coordinates its children’s state changes as explained later. It
152 also distributes events and various other functionality to elements.
154 A bin can have its own source and sinkpads by ghostpadding one or more
155 of its children’s pads to itself.
157 Below is a picture of a bin with two elements. The sinkpad of one
158 element is ghostpadded to the bin.
161 +---------------------------+
163 | +--------+ +-------+ |
166 sink +--------+ +-------+ |
167 +---------------------------+
172 A pipeline is a special bin subclass that provides the following
173 features to its children:
175 - Select and manage a global clock for all its children.
176 - Manage `running_time` based on the selected clock. Running\_time is
177 the elapsed time the pipeline spent in the `PLAYING` state and is used
179 - Manage latency in the pipeline.
180 - Provide means for elements to comunicate with the application by the
182 - Manage the global state of the elements such as Errors and
185 Normally the application creates one pipeline that will manage all the
186 elements in the application.
188 ## Dataflow and buffers
190 GStreamer supports two possible types of dataflow, the push and pull
191 model. In the push model, an upstream element sends data to a downstream
192 element by calling a method on a sinkpad. In the pull model, a
193 downstream element requests data from an upstream element by calling a
194 method on a source pad.
196 The most common dataflow is the push model. The pull model can be used
197 in specific circumstances by demuxer elements. The pull model can also
198 be used by low latency audio applications.
200 The data passed between pads is encapsulated in Buffers. The buffer
201 contains pointers to the actual memory and also metadata describing the
202 memory. This metadata includes:
204 - timestamp of the data, this is the time instance at which the data
205 was captured or the time at which the data should be played back.
207 - offset of the data: a media specific offset, this could be samples
208 for audio or frames for video.
210 - the duration of the data in time.
212 - additional flags describing special properties of the data such as
213 discontinuities or delta units.
215 - additional arbitrary metadata
217 When an element whishes to send a buffer to another element is does this
218 using one of the pads that is linked to a pad of the other element. In
219 the push model, a buffer is pushed to the peer pad with
220 `gst_pad_push()`. In the pull model, a buffer is pulled from the peer
221 with the `gst_pad_pull_range()` function.
223 Before an element pushes out a buffer, it should make sure that the peer
224 element can understand the buffer contents. It does this by querying the
225 peer element for the supported formats and by selecting a suitable
226 common format. The selected format is then first sent to the peer
227 element with a CAPS event before pushing the buffer (see
228 [negotiation](design/negotiation.md)).
230 When an element pad receives a CAPS event, it has to check if it
231 understand the media type. The element must refuse following buffers if
232 the media type preceding it was not accepted.
234 Both `gst_pad_push()` and `gst_pad_pull_range()` have a return value
235 indicating whether the operation succeeded. An error code means that no
236 more data should be sent to that pad. A source element that initiates
237 the data flow in a thread typically pauses the producing thread when
240 A buffer can be created with `gst_buffer_new()` or by requesting a
241 usable buffer from a buffer pool using
242 `gst_buffer_pool_acquire_buffer()`. Using the second method, it is
243 possible for the peer element to implement a custom buffer allocation
246 The process of selecting a media type is called caps negotiation.
250 A media type (Caps) is described using a generic list of key/value
251 pairs. The key is a string and the value can be a single/list/range of
254 Caps that have no ranges/list or other variable parts are said to be
255 fixed and can be used to put on a buffer.
257 Caps with variables in them are used to describe possible media types
258 that can be handled by a pad.
260 ## Dataflow and events
262 Parallel to the dataflow is a flow of events. Unlike the buffers, events
263 can pass both upstream and downstream. Some events only travel upstream
264 others only downstream.
266 The events are used to denote special conditions in the dataflow such as
267 EOS or to inform plugins of special events such as flushing or seeking.
269 Some events must be serialized with the buffer flow, others don’t.
270 Serialized events are inserted between the buffers. Non serialized
271 events jump in front of any buffers current being processed.
273 An example of a serialized event is a TAG event that is inserted between
274 buffers to mark metadata for those buffers.
276 An example of a non serialized event is the FLUSH event.
278 ## Pipeline construction
280 The application starts by creating a Pipeline element using
281 `gst_pipeline_new()`. Elements are added to and removed from the
282 pipeline with `gst_bin_add()` and `gst_bin_remove()`.
284 After adding the elements, the pads of an element can be retrieved with
285 `gst_element_get_pad()`. Pads can then be linked together with
288 Some elements create new pads when actual dataflow is happening in the
289 pipeline. With `g_signal_connect()` one can receive a notification when
290 an element has created a pad. These new pads can then be linked to other
293 Some elements cannot be linked together because they operate on
294 different incompatible data types. The possible datatypes a pad can
295 provide or consume can be retrieved with `gst_pad_get_caps()`.
297 Below is a simple mp3 playback pipeline that we constructed. We will use
298 this pipeline in further examples.
301 +-------------------------------------------+
303 | +---------+ +----------+ +----------+ |
304 | | filesrc | | mp3dec | | alsasink | |
305 | | src-sink src-sink | |
306 | +---------+ +----------+ +----------+ |
307 +-------------------------------------------+
312 One of the important functions of the pipeline is to select a global
313 clock for all the elements in the pipeline.
315 The purpose of the clock is to provide a stricly increasing value at the
316 rate of one `GST_SECOND` per second. Clock values are expressed in
317 nanoseconds. Elements use the clock time to synchronize the playback of
320 Before the pipeline is set to `PLAYING`, the pipeline asks each element if
321 they can provide a clock. The clock is selected in the following order:
323 - If the application selected a clock, use that one.
325 - If a source element provides a clock, use that clock.
327 - Select a clock from any other element that provides a clock, start
330 - If no element provides a clock a default system clock is used for
333 In a typical playback pipeline this algorithm will select the clock
334 provided by a sink element such as an audio sink.
336 In capture pipelines, this will typically select the clock of the data
337 producer, which in most cases can not control the rate at which it
342 When all the pads are linked and signals have been connected, the
343 pipeline can be put in the `PAUSED` state to start dataflow.
345 When a bin (and hence a pipeline) performs a state change, it will
346 change the state of all its children. The pipeline will change the state
347 of its children from the sink elements to the source elements, this to
348 make sure that no upstream element produces data to an element that is
349 not yet ready to accept it.
351 In the mp3 playback pipeline, the state of the elements is changed in
352 the order alsasink, mp3dec, filesrc.
354 All intermediate states are traversed for each element resulting in the
355 following chain of state changes:
357 * alsasink to `READY`: the audio device is probed
359 * mp3dec to `READY`: nothing happens
361 * filesrc to `READY`: the file is probed
363 * alsasink to `PAUSED`: the audio device is opened. alsasink is a sink and returns `ASYNC` because it did not receive data yet
365 * mp3dec to `PAUSED`: the decoding library is initialized
367 * filesrc to `PAUSED`: the file is opened and a thread is started to push data to mp3dec
369 At this point data flows from filesrc to mp3dec and alsasink. Since
370 mp3dec is `PAUSED`, it accepts the data from filesrc on the sinkpad and
371 starts decoding the compressed data to raw audio samples.
373 The mp3 decoder figures out the samplerate, the number of channels and
374 other audio properties of the raw audio samples and sends out a caps
375 event with the media type.
377 Alsasink then receives the caps event, inspects the caps and
378 reconfigures itself to process the media type.
380 mp3dec then puts the decoded samples into a Buffer and pushes this
381 buffer to the next element.
383 Alsasink receives the buffer with samples. Since it received the first
384 buffer of samples, it completes the state change to the PAUSED state. At
385 this point the pipeline is prerolled and all elements have samples.
386 Alsasink is now also capable of providing a clock to the pipeline.
388 Since alsasink is now in the `PAUSED` state it blocks while receiving the
389 first buffer. This effectively blocks both mp3dec and filesrc in their
392 Since all elements now return `SUCCESS` from the
393 `gst_element_get_state()` function, the pipeline can be put in the
396 Before going to `PLAYING`, the pipeline select a clock and samples the
397 current time of the clock. This is the `base_time`. It then distributes
398 this time to all elements. Elements can then synchronize against the
399 clock using the buffer `running_time`
400 `base_time` (See also [synchronisation](design/synchronisation.md)).
402 The following chain of state changes then takes place:
404 * alsasink to `PLAYING`: the samples are played to the audio device
406 * mp3dec to `PLAYING`: nothing happens
408 * filesrc to `PLAYING`: nothing happens
412 The pipeline informs the application of any special events that occur in
413 the pipeline with the bus. The bus is an object that the pipeline
414 provides and that can be retrieved with `gst_pipeline_get_bus()`.
416 The bus can be polled or added to the glib mainloop.
418 The bus is distributed to all elements added to the pipeline. The
419 elements use the bus to post messages on. Various message types exist
420 such as `ERRORS`, `WARNINGS`, `EOS`, `STATE_CHANGED`, etc..
422 The pipeline handles `EOS` messages received from elements in a special
423 way. It will only forward the message to the application when all sink
424 elements have posted an `EOS` message.
426 Other methods for obtaining the pipeline status include the Query
427 functionality that can be performed with `gst_element_query()` on the
428 pipeline. This type of query is useful for obtaining information about
429 the current position and total time of the pipeline. It can also be used
430 to query for the supported seeking formats and ranges.
434 When the source filter encounters the end of the stream, it sends an EOS
435 event to the peer element. This event will then travel downstream to all
436 of the connected elements to inform them of the EOS. The element is not
437 supposed to accept any more data after receiving an EOS event on a
440 The element providing the streaming thread stops sending data after
441 sending the `EOS` event.
443 The EOS event will eventually arrive in the sink element. The sink will
444 then post an `EOS` message on the bus to inform the pipeline that a
445 particular stream has finished. When all sinks have reported `EOS`, the
446 pipeline forwards the EOS message to the application. The `EOS` message is
447 only forwarded to the application in the `PLAYING` state.
449 When in `EOS`, the pipeline remains in the `PLAYING` state, it is the
450 applications responsability to `PAUSE` or `READY` the pipeline. The
451 application can also issue a seek, for example.
455 When a running pipeline is set from the `PLAYING` to `READY` state, the
456 following actions occur in the pipeline:
458 * alsasink to `PAUSED`: alsasink blocks and completes the state change on the
459 next sample. If the element was `EOS`, it does not wait for a sample to complete
461 * mp3dec to `PAUSED`: nothing
462 * filesrc to `PAUSED`: nothing
464 Going to the intermediate `PAUSED` state will block all elements in the
465 `_push()` functions. This happens because the sink element blocks on the
466 first buffer it receives.
468 Some elements might be performing blocking operations in the `PLAYING`
469 state that must be unblocked when they go into the PAUSED state. This
470 makes sure that the state change happens very fast.
472 In the next `PAUSED` to `READY` state change the pipeline has to shut down
473 and all streaming threads must stop sending data. This happens in the
476 * alsasink to `READY`: alsasink unblocks from the `_chain()` function and returns
477 a `FLUSHING` return value to the peer element. The sinkpad is deactivated and
478 becomes unusable for sending more data.
479 * mp3dec to `READY`: the pads are deactivated and the state change completes
480 when mp3dec leaves its `_chain()` function.
481 * filesrc to `READY`: the pads are deactivated and the thread is paused.
483 The upstream elements finish their `_chain()` function because the
484 downstream element returned an error code (`FLUSHING`) from the `_push()`
485 functions. These error codes are eventually returned to the element that
486 started the streaming thread (filesrc), which pauses the thread and
487 completes the state change.
489 This sequence of events ensure that all elements are unblocked and all
490 streaming threads stopped.
494 Seeking in the pipeline requires a very specific order of operations to
495 make sure that the elements remain synchronized and that the seek is
496 performed with a minimal amount of latency.
498 An application issues a seek event on the pipeline using
499 `gst_element_send_event()` on the pipeline element. The event can be a
500 seek event in any of the formats supported by the elements.
502 The pipeline first pauses the pipeline to speed up the seek operations.
504 The pipeline then issues the seek event to all sink elements. The sink
505 then forwards the seek event upstream until some element can perform the
506 seek operation, which is typically the source or demuxer element. All
507 intermediate elements can transform the requested seek offset to another
508 format, this way a decoder element can transform a seek to a frame
509 number to a timestamp, for example.
511 When the seek event reaches an element that will perform the seek
512 operation, that element performs the following steps.
514 1) send a `FLUSH_START` event to all downstream and upstream peer elements.
515 2) make sure the streaming thread is not running. The streaming thread will
516 always stop because of step 1).
517 3) perform the seek operation
518 4) send a `FLUSH` done event to all downstream and upstream peer elements.
519 5) send `SEGMENT` event to inform all elements of the new position and to complete
522 In step 1) all downstream elements have to return from any blocking
523 operations and have to refuse any further buffers or events different
526 The first step ensures that the streaming thread eventually unblocks and
527 that step 2) can be performed. At this point, dataflow is completely
528 stopped in the pipeline.
530 In step 3) the element performs the seek to the requested position.
532 In step 4) all peer elements are allowed to accept data again and
533 streaming can continue from the new position. A FLUSH done event is sent
534 to all the peer elements so that they accept new data again and restart
535 their streaming threads.
537 Step 5) informs all elements of the new position in the stream. After
538 that the event function returns back to the application. and the
539 streaming threads start to produce new data.
541 Since the pipeline is still `PAUSED`, this will preroll the next media
542 sample in the sinks. The application can wait for this preroll to
543 complete by performing a `_get_state()` on the pipeline.
545 The last step in the seek operation is then to adjust the stream
546 `running_time` of the pipeline to 0 and to set the pipeline back to
549 The sequence of events in our mp3 playback example.
552 | a) seek on pipeline
554 +----------------------------------V--------+
555 | pipeline | c) seek on sink
556 | +---------+ +----------+ +---V------+ |
557 | | filesrc | | mp3dec | | alsasink | |
558 | | src-sink src-sink | |
559 | +---------+ +----------+ +----|-----+ |
560 +-----------------------------------|-------+
561 <------------------------+
562 d) seek travels upstream
564 --------------------------> 1) FLUSH event
567 --------------------------> 4) FLUSH done event
568 --------------------------> 5) SEGMENT event
570 | e) update running_time to 0