4 This part gives an overview of the design of GStreamer with references to
5 the more detailed explanations of the different topics.
7 This document is intented for people that want to have a global overview of
8 the inner workings of GStreamer.
14 GStreamer is a set of libraries and plugins that can be used to implement various
15 multimedia applications ranging from desktop players, audio/video recorders,
16 multimedia servers, transcoders, etc.
18 Applications are built by constructing a pipeline composed of elements. An element
19 is an object that performs some action on a multimedia stream such as:
22 - decode or encode between formats
23 - capture from a hardware device
24 - render to a hardware device
25 - mix or multiplex multiple streams
27 Elements have input and output pads called sink and source pads in GStreamer. An
28 application links elements together on pads to construct a pipeline. Below is
29 an example of an ogg/vorbis playback pipeline.
31 +-----------------------------------------------------------+
32 | ----------> downstream -------------------> |
35 | +---------+ +----------+ +-----------+ +----------+ |
36 | | filesrc | | oggdemux | | vorbisdec | | alsasink | |
37 | | src-sink src-sink src-sink | |
38 | +---------+ +----------+ +-----------+ +----------+ |
40 | <---------< upstream <-------------------< |
41 +-----------------------------------------------------------+
43 The filesrc element reads data from a file on disk. The oggdemux element parses
44 the data and sends the compressed audio data to the vorbisdec element. The
45 vorbisdec element decodes the compressed data and sends it to the alsasink
46 element. The alsasink element sends the samples to the audio card for playback.
48 Downstream and upstream are the terms used to describe the direction in the
49 Pipeline. From source to sink is called "downstream" and "upstream" is
50 from sink to source. Dataflow always happens downstream.
52 The task of the application is to construct a pipeline as above using existing
53 elements. This is further explained in the pipeline building topic.
55 The application does not have to manage any of the complexities of the
56 actual dataflow/decoding/conversions/synchronisation etc. but only calls high
57 level functions on the pipeline object such as PLAY/PAUSE/STOP.
59 The application also receives messages and notifications from the pipeline such
60 as metadata, warning, error and EOS messages.
62 If the application needs more control over the graph it is possible to directly
63 access the elements and pads in the pipeline.
69 GStreamer design goals include:
71 - Process large amounts of data quickly
72 - Allow fully multithreaded processing
73 - Ability to deal with multiple formats
74 - Synchronize different dataflows
75 - Ability to deal with multiple devices
77 The capabilities presented to the application depends on the number of elements
78 installed on the system and their functionality.
80 The GStreamer core is designed to be media agnostic but provides many features
81 to elements to describe media formats.
87 The smallest building blocks in a pipeline are elements. An element provides a
88 number of pads which can be source or sinkpads. Sourcepads provide data and
89 sinkpads consume data. Below is an example of an ogg demuxer element that has
90 one pad that takes (sinks) data and two source pads that produce data.
98 An element can be in four different states: NULL, READY, PAUSED, PLAYING. In the
99 NULL and READY state, the element is not processing any data. In the PLAYING state
100 it is processing data. The intermediate PAUSED state is used to preroll data in
101 the pipeline. A state change can be performed with gst_element_set_state().
103 An element always goes through all the intermediate state changes. This means that
104 when en element is in the READY state and is put to PLAYING, it will first go
105 through the intermediate PAUSED state.
107 An element state change to PAUSED will activate the pads of the element. First the
108 source pads are activated, then the sinkpads. When the pads are activated, the
109 pad activate function is called. Some pads will start a thread (GstTask) or some
110 other mechanism to start producing or consuming data.
112 The PAUSED state is special as it is used to preroll data in the pipeline. The purpose
113 is to fill all connected elements in the pipeline with data so that the subsequent
114 PLAYING state change happens very quickly. Some elements will therefore not complete
115 the state change to PAUSED before they have received enough data. Sink elements are
116 required to only complete the state change to PAUSED after receiving the first data.
118 Normally the state changes of elements are coordinated by the pipeline as explained
119 in [part-states.txt].
121 Different categories of elements exist:
123 - source elements, these are elements that do not consume data but only provide data
125 - sink elements, these are elements that do not produce data but renders data to
127 - transform elements, these elements transform an input stream in a certain format
128 into a stream of another format. Encoder/decoder/converters are examples.
129 - demuxer elements, these elements parse a stream and produce several output streams.
130 - mixer/muxer elements, combine several input streams into one output stream.
132 Other categories of elements can be constructed (see part-klass.txt).
138 A bin is an element subclass and acts as a container for other elements so that multiple
139 elements can be combined into one element.
141 A bin coordinates its children's state changes as explained later. It also distributes
142 events and various other functionality to elements.
144 A bin can have its own source and sinkpads by ghostpadding one or more of its children's
147 Below is a picture of a bin with two elements. The sinkpad of one element is ghostpadded
150 +---------------------------+
152 | +--------+ +-------+ |
155 sink +--------+ +-------+ |
156 +---------------------------+
162 A pipeline is a special bin subclass that provides the following features to its
165 - Select and manage a global clock for all its children.
166 - Manage running_time based on the selected clock. Running_time is the elapsed
167 time the pipeline spent in the PLAYING state and is used for
169 - Manage latency in the pipeline.
170 - Provide means for elements to comunicate with the application by the GstBus.
171 - Manage the global state of the elements such as Errors and end-of-stream.
173 Normally the application creates one pipeline that will manage all the elements
180 GStreamer supports two possible types of dataflow, the push and pull model. In the
181 push model, an upstream element sends data to a downstream element by calling a
182 method on a sinkpad. In the pull model, a downstream element requests data from
183 an upstream element by calling a method on a source pad.
185 The most common dataflow is the push model. The pull model can be used in specific
186 circumstances by demuxer elements. The pull model can also be used by low latency
189 The data passed between pads is encapsulated in Buffers. The buffer contains a
190 pointer to the actual data and also metadata describing the data. This metadata
193 - timestamp of the data, this is the time instance at which the data was captured
194 or the time at which the data should be played back.
195 - offset of the data: a media specific offset, this could be samples for audio or
197 - the duration of the data in time.
198 - additional flags describing special properties of the data such as
199 discontinuities or delta units.
200 - additional arbitrary metadata
202 When an element whishes to send a buffer to another element is does this using one
203 of the pads that is linked to a pad of the other element. In the push model, a
204 buffer is pushed to the peer pad with gst_pad_push(). In the pull model, a buffer
205 is pulled from the peer with the gst_pad_pull_range() function.
207 Before an element pushes out a buffer, it should make sure that the peer element
208 can understand the buffer contents. It does this by querying the peer element
209 for the supported formats and by selecting a suitable common format. The selected
210 format is then first sent to the peer element with a CAPS event before pushing
214 When an element pad receives a CAPS event, it has to check if it understand the
215 media type. The element must refuse following buffers if the media type
216 preceeding it was not accepted.
218 Both gst_pad_push() and gst_pad_pull_range() have a return value indicating whether
219 the operation succeeded. An error code means that no more data should be sent
220 to that pad. A source element that initiates the data flow in a thread typically
221 pauses the producing thread when this happens.
223 A buffer can be created with gst_buffer_new() or by requesting a usable buffer
224 from a buffer pool using gst_buffer_pool_acquire_buffer(). Using the second
225 method, it is possible for the peer element to implement a custom buffer
226 allocation algorithm.
228 The process of selecting a media type is called caps negotiation.
234 A media type (Caps) is described using a generic list of key/value pairs. The key is
235 a string and the value can be a single/list/range of int/float/string.
237 Caps that have no ranges/list or other variable parts are said to be fixed and
238 can be used to put on a buffer.
240 Caps with variables in them are used to describe possible media types that can be
247 Parallel to the dataflow is a flow of events. Unlike the buffers, events can pass
248 both upstream and downstream. Some events only travel upstream others only downstream.
250 The events are used to denote special conditions in the dataflow such as EOS or
251 to inform plugins of special events such as flushing or seeking.
253 Some events must be serialized with the buffer flow, others don't. Serialized
254 events are inserted between the buffers. Non serialized events jump in front
255 of any buffers current being processed.
257 An example of a serialized event is a TAG event that is inserted between buffers
258 to mark metadata for those buffers.
260 An example of a non serialized event is the FLUSH event.
263 Pipeline construction
264 ~~~~~~~~~~~~~~~~~~~~~
266 The application starts by creating a Pipeline element using gst_pipeline_new ().
267 Elements are added to and removed from the pipeline with gst_bin_add() and
270 After adding the elements, the pads of an element can be retrieved with
271 gst_element_get_pad(). Pads can then be linked together with gst_pad_link().
273 Some elements create new pads when actual dataflow is happening in the pipeline.
274 With g_signal_connect() one can receive a notification when an element has created
275 a pad. These new pads can then be linked to other unlinked pads.
277 Some elements cannot be linked together because they operate on different
278 incompatible data types. The possible datatypes a pad can provide or consume can
279 be retrieved with gst_pad_get_caps().
281 Below is a simple mp3 playback pipeline that we constructed. We will use this
282 pipeline in further examples.
284 +-------------------------------------------+
286 | +---------+ +----------+ +----------+ |
287 | | filesrc | | mp3dec | | alsasink | |
288 | | src-sink src-sink | |
289 | +---------+ +----------+ +----------+ |
290 +-------------------------------------------+
296 One of the important functions of the pipeline is to select a global clock
297 for all the elements in the pipeline.
299 The purpose of the clock is to provide a stricly increasing value at the rate
300 of one GST_SECOND per second. Clock values are expressed in nanoseconds.
301 Elements use the clock time to synchronize the playback of data.
303 Before the pipeline is set to PLAYING, the pipeline asks each element if they can
304 provide a clock. The clock is selected in the following order:
306 - If the application selected a clock, use that one.
307 - If a source element provides a clock, use that clock.
308 - Select a clock from any other element that provides a clock, start with the
310 - If no element provides a clock a default system clock is used for the pipeline.
312 In a typical playback pipeline this algorithm will select the clock provided by
313 a sink element such as an audio sink.
315 In capture pipelines, this will typically select the clock of the data producer, which
316 in most cases can not control the rate at which it produces data.
322 When all the pads are linked and signals have been connected, the pipeline can
323 be put in the PAUSED state to start dataflow.
325 When a bin (and hence a pipeline) performs a state change, it will change the state
326 of all its children. The pipeline will change the state of its children from the
327 sink elements to the source elements, this to make sure that no upstream element
328 produces data to an element that is not yet ready to accept it.
330 In the mp3 playback pipeline, the state of the elements is changed in the order
331 alsasink, mp3dec, filesrc.
333 All intermediate states are traversed for each element resulting in the following
334 chain of state changes:
336 alsasink to READY: the audio device is probed
337 mp3dec to READY: nothing happens.
338 filesrc to READY: the file is probed
339 alsasink to PAUSED: the audio device is opened. alsasink is a sink and returns
340 ASYNC because it did not receive data yet.
341 mp3dec to PAUSED: the decoding library is initialized
342 filesrc to PAUSED: the file is opened and a thread is started to push data to
345 At this point data flows from filesrc to mp3dec and alsasink. Since mp3dec is PAUSED,
346 it accepts the data from filesrc on the sinkpad and starts decoding the compressed
347 data to raw audio samples.
349 The mp3 decoder figures out the samplerate, the number of channels and other audio
350 properties of the raw audio samples and sends out a caps event with the media type.
352 Alsasink then receives the caps event, inspects the caps and reconfigures
353 itself to process the media type.
355 mp3dec then puts the decoded samples into a Buffer and pushes this buffer to the next
358 Alsasink receives the buffer with samples. Since it received the first buffer of
359 samples, it completes the state change to the PAUSED state. At this point the
360 pipeline is prerolled and all elements have samples. Alsasink is now also
361 capable of providing a clock to the pipeline.
363 Since alsasink is now in the PAUSED state it blocks while receiving the first buffer. This
364 effectively blocks both mp3dec and filesrc in their gst_pad_push().
366 Since all elements now return SUCCESS from the gst_element_get_state() function,
367 the pipeline can be put in the PLAYING state.
369 Before going to PLAYING, the pipeline select a clock and samples the current time of
370 the clock. This is the base_time. It then distributes this time to all elements.
371 Elements can then synchronize against the clock using the buffer running_time +
372 base_time (See also part-synchronisation.txt).
374 The following chain of state changes then takes place:
376 alsasink to PLAYING: the samples are played to the audio device
377 mp3dec to PLAYING: nothing happens
378 filesrc to PLAYING: nothing happens
384 The pipeline informs the application of any special events that occur in the
385 pipeline with the bus. The bus is an object that the pipeline provides and that
386 can be retrieved with gst_pipeline_get_bus().
388 The bus can be polled or added to the glib mainloop.
390 The bus is distributed to all elements added to the pipeline. The elements use the bus
391 to post messages on. Various message types exist such as ERRORS, WARNINGS, EOS,
394 The pipeline handles EOS messages received from elements in a special way. It will
395 only forward the message to the application when all sink elements have posted an
398 Other methods for obtaining the pipeline status include the Query functionality that
399 can be performed with gst_element_query() on the pipeline. This type of query
400 is useful for obtaining information about the current position and total time of
401 the pipeline. It can also be used to query for the supported seeking formats and
408 When the source filter encounters the end of the stream, it sends an EOS event to
409 the peer element. This event will then travel downstream to all of the connected
410 elements to inform them of the EOS. The element is not supposed to accept any more
411 data after receiving an EOS event on a sinkpad.
413 The element providing the streaming thread stops sending data after sending the
416 The EOS event will eventually arrive in the sink element. The sink will then post
417 an EOS message on the bus to inform the pipeline that a particular stream has
418 finished. When all sinks have reported EOS, the pipeline forwards the EOS message
419 to the application. The EOS message is only forwarded to the application in the
422 When in EOS, the pipeline remains in the PLAYING state, it is the applications
423 responsability to PAUSE or READY the pipeline. The application can also issue
430 When a running pipeline is set from the PLAYING to READY state, the following
431 actions occur in the pipeline:
433 alsasink to PAUSED: alsasink blocks and completes the state change on the
434 next sample. If the element was EOS, it does not wait for
435 a sample to complete the state change.
436 mp3dec to PAUSED: nothing
437 filesrc to PAUSED: nothing
439 Going to the intermediate PAUSED state will block all elements in the _push()
440 functions. This happens because the sink element blocks on the first buffer
443 Some elements might be performing blocking operations in the PLAYING state that
444 must be unblocked when they go into the PAUSED state. This makes sure that the
445 state change happens very fast.
447 In the next PAUSED to READY state change the pipeline has to shut down and all
448 streaming threads must stop sending data. This happens in the following sequence:
450 alsasink to READY: alsasink unblocks from the _chain() function and returns a
451 WRONG_STATE return value to the peer element. The sinkpad is
452 deactivated and becomes unusable for sending more data.
453 mp3dec to READY: the pads are deactivated and the state change completes when
454 mp3dec leaves its _chain() function.
455 filesrc to READY: the pads are deactivated and the thread is paused.
457 The upstream elements finish their chain() function because the downstream element
458 returned an error code (WRONG_STATE) from the _push() functions. These error codes
459 are eventually returned to the element that started the streaming thread (filesrc),
460 which pauses the thread and completes the state change.
462 This sequence of events ensure that all elements are unblocked and all streaming
469 Seeking in the pipeline requires a very specific order of operations to make
470 sure that the elements remain synchronized and that the seek is performed with
471 a minimal amount of latency.
473 An application issues a seek event on the pipeline using gst_element_send_event()
474 on the pipeline element. The event can be a seek event in any of the formats
475 supported by the elements.
477 The pipeline first pauses the pipeline to speed up the seek operations.
479 The pipeline then issues the seek event to all sink elements. The sink then forwards
480 the seek event upstream until some element can perform the seek operation, which is
481 typically the source or demuxer element. All intermediate elements can transform the
482 requested seek offset to another format, this way a decoder element can transform a
483 seek to a frame number to a timestamp, for example.
485 When the seek event reaches an element that will perform the seek operation, that
486 element performs the following steps.
488 1) send a FLUSH_START event to all downstream and upstream peer elements.
489 2) make sure the streaming thread is not running. The streaming thread will
490 always stop because of step 1).
491 3) perform the seek operation
492 4) send a FLUSH done event to all downstream and upstream peer elements.
493 5) send SEGMENT event to inform all elements of the new position and to complete
496 In step 1) all downstream elements have to return from any blocking operations
497 and have to refuse any further buffers or events different from a FLUSH done.
499 The first step ensures that the streaming thread eventually unblocks and that
500 step 2) can be performed. At this point, dataflow is completely stopped in the
503 In step 3) the element performs the seek to the requested position.
505 In step 4) all peer elements are allowed to accept data again and streaming
506 can continue from the new position. A FLUSH done event is sent to all the peer
507 elements so that they accept new data again and restart their streaming threads.
509 Step 5) informs all elements of the new position in the stream. After that the
510 event function returns back to the application. and the streaming threads start
513 Since the pipeline is still PAUSED, this will preroll the next media sample in the
514 sinks. The application can wait for this preroll to complete by performing a
515 _get_state() on the pipeline.
517 The last step in the seek operation is then to adjust the stream running_time of
518 the pipeline to 0 and to set the pipeline back to PLAYING.
520 The sequence of events in our mp3 playback example.
522 | a) seek on pipeline
524 +----------------------------------V--------+
525 | pipeline | c) seek on sink
526 | +---------+ +----------+ +---V------+ |
527 | | filesrc | | mp3dec | | alsasink | |
528 | | src-sink src-sink | |
529 | +---------+ +----------+ +----|-----+ |
530 +-----------------------------------|-------+
531 <------------------------+
532 d) seek travels upstream
534 --------------------------> 1) FLUSH event
537 --------------------------> 4) FLUSH done event
538 --------------------------> 5) SEGMENT event
540 | e) update running_time to 0