1 <chapter id="chapter-negotiation" xreflabel="Caps negotiation">
2 <title>Caps negotiation</title>
4 Caps negotiation is the act of finding a media format (GstCaps) between
5 elements that they can handle. This process in &GStreamer; can in most
6 cases find an optimal solution for the complete pipeline. In this section
7 we explain how this works.
10 <sect1 id="section-nego-basics">
11 <title>Caps negotiation basics</title>
13 In &GStreamer;, negotiation of the media format always follows the
14 following simple rules:
19 A downstream element suggest a format on its sinkpad and places the
20 suggestion in the result of the CAPS query performed on the sinkpad.
21 See also <xref linkend="section-nego-getcaps"/>.
26 An upstream element decides on a format. It sends the selected media
27 format downstream on its source pad with a CAPS event. Downstream
28 elements reconfigure themselves to handle the media type in the CAPS
34 A downstream element can inform upstream that it would like to
35 suggest a new format by sending a RECONFIGURE event upstream. The
36 RECONFIGURE event simply instructs an upstream element to restart
37 the negotiation phase. Because the element that sent out the
38 RECONFIGURE event is now suggesting another format, the format
39 in the pipeline might change.
44 In addition to the CAPS and RECONFIGURE event and the CAPS query, there
45 is an ACCEPT_CAPS query to quickly check if a certain caps can
46 be accepted by an element.
49 All negotiation follows these simple rules. Let's take a look at some
50 typical uses cases and how negotiation happens.
54 <sect1 id="section-nego-usecases">
55 <title>Caps negotiation use cases</title>
57 In what follows we will look at some use cases for push-mode scheduling.
58 The pull-mode scheduling negotiation phase is discussed in
59 <xref linkend="section-nego-pullmode"/> and is actually similar as we
63 Since the sink pads only suggest formats and the source pads need to
64 decide, the most complicated work is done in the source pads.
65 We can identify 3 caps negotiation use cases for the source pads:
70 Fixed negotiation. An element can output one format only.
71 See <xref linkend="section-nego-fixed"/>.
76 Transform negotiation. There is a (fixed) transform between the
77 input and output format of the element, usually based on some
78 element property. The caps that the element will produce depend
79 on the upstream caps and the caps that the element can accept
80 depend on the downstream caps.
81 See <xref linkend="section-nego-transform"/>.
86 Dynamic negotiation. An element can output many formats.
87 See <xref linkend="section-nego-dynamic"/>.
92 <sect2 id="section-nego-fixed">
93 <title>Fixed negotiation</title>
95 In this case, the source pad can only produce a fixed format. Usually
96 this format is encoded inside the media. No downstream element can
97 ask for a different format, the only way that the source pad will
98 renegotiate is when the element decides to change the caps itself.
101 Elements that could implement fixed caps (on their source pads) are,
102 in general, all elements that are not renegotiable. Examples include:
107 A typefinder, since the type found is part of the actual data stream
108 and can thus not be re-negotiated. The typefinder will look at the
109 stream of bytes, figure out the type, send a CAPS event with the
110 caps and then push buffers of the type.
115 Pretty much all demuxers, since the contained elementary data
116 streams are defined in the file headers, and thus not
122 Some decoders, where the format is embedded in the data stream
123 and not part of the peercaps <emphasis>and</emphasis> where the
124 decoder itself is not reconfigurable, too.
129 Some sources that produce a fixed format.
134 <function>gst_pad_use_fixed_caps()</function> is used on the source
135 pad with fixed caps. As long as the pad is not negotiated, the default
136 CAPS query will return the caps presented in the padtemplate. As soon
137 as the pad is negotiated, the CAPS query will return the negotiated
138 caps (and nothing else). These are the relevant code snippets for fixed
144 pad = gst_pad_new_from_static_template (..);
145 gst_pad_use_fixed_caps (pad);
150 The fixed caps can then be set on the pad by calling
151 <function>gst_pad_set_caps ()</function>.
156 caps = gst_caps_new_simple ("audio/x-raw",
157 "format", G_TYPE_STRING, GST_AUDIO_NE(F32),
158 "rate", G_TYPE_INT, <samplerate>,
159 "channels", G_TYPE_INT, <num-channels>, NULL);
160 if (!gst_pad_set_caps (pad, caps)) {
161 GST_ELEMENT_ERROR (element, CORE, NEGOTIATION, (NULL),
162 ("Some debug information here"));
163 return GST_FLOW_ERROR;
169 These types of elements also don't have a relation between the input
170 format and the output format, the input caps simply don't contain the
171 information needed to produce the output caps.
174 All other elements that need to be configured for the format should
175 implement full caps negotiation, which will be explained in the next
180 <sect2 id="section-nego-transform">
181 <title>Transform negotiation</title>
183 In this negotiation technique, there is a fixed transform between
184 the element input caps and the output caps. This transformation
185 could be parameterized by element properties but not by the
186 content of the stream (see <xref linkend="section-nego-fixed"/>
190 The caps that the element can accept depend on the (fixed
191 transformation) downstream caps. The caps that the element can
192 produce depend on the (fixed transformation of) the upstream
196 This type of element can usually set caps on its source pad from
197 the <function>_event()</function> function on the sink pad when
198 it received the CAPS event. This means that the caps transform
199 function transforms a fixed caps into another fixed caps.
200 Examples of elements include:
205 Videobox. It adds configurable border around a video frame
206 depending on object properties.
211 Identity elements. All elements that don't change the format
212 of the data, only the content. Video and audio effects are an
213 example. Other examples include elements that inspect the
219 Some decoders and encoders, where the output format is defined
220 by input format, like mulawdec and mulawenc. These decoders
221 usually have no headers that define the content of the stream.
222 They are usually more like conversion elements.
227 Below is an example of a negotiation steps of a typical transform
228 element. In the sink pad CAPS event handler, we compute the caps
229 for the source pad and set those.
236 gst_my_filter_setcaps (GstMyFilter *filter,
239 GstStructure *structure;
244 structure = gst_caps_get_structure (caps, 0);
245 ret = gst_structure_get_int (structure, "rate", &rate);
246 ret = ret && gst_structure_get_int (structure, "channels", &channels);
250 outcaps = gst_caps_new_simple ("audio/x-raw",
251 "format", G_TYPE_STRING, GST_AUDIO_NE(S16),
252 "rate", G_TYPE_INT, samplerate,
253 "channels", G_TYPE_INT, channels, NULL);
254 ret = gst_pad_set_caps (filter->srcpad, outcaps);
255 gst_caps_unref (outcaps);
261 gst_my_filter_sink_event (GstPad *pad,
266 GstMyFilter *filter = GST_MY_FILTER (parent);
268 switch (GST_EVENT_TYPE (event)) {
273 gst_event_parse_caps (event, &caps);
274 ret = gst_my_filter_setcaps (filter, caps);
278 ret = gst_pad_event_default (pad, parent, event);
289 <sect2 id="section-nego-dynamic">
290 <title>Dynamic negotiation</title>
292 A last negotiation method is the most complex and powerful dynamic
296 Like with the transform negotiation in
297 <xref linkend="section-nego-transform"/>, dynamic negotiation will
298 perform a transformation on the downstream/upstream caps. Unlike the
299 transform negotiation, this transform will convert fixed caps to
300 unfixed caps. This means that the sink pad input caps can be converted
301 into unfixed (multiple) formats. The source pad will have to choose a
302 format from all the possibilities. It would usually like to choose a
303 format that requires the least amount of effort to produce but it does
304 not have to be. The selection of the format should also depend on the
305 caps that can be accepted downstream (see a QUERY_CAPS function in
306 <xref linkend="section-nego-getcaps"/>).
309 A typical flow goes like this:
314 Caps are received on the sink pad of the element.
319 If the element prefers to operate in passthrough mode, check
320 if downstream accepts the caps with the ACCEPT_CAPS query. If it
321 does, we can complete negotiation and we can operate in
327 Calculate the possible caps for the source pad.
332 Query the downstream peer pad for the list of possible
338 Select from the downstream list the first caps that you can
339 transform to and set this as the output caps. You might have to
340 fixate the caps to some reasonable defaults to construct
346 Examples of this type of elements include:
351 Converter elements such as videoconvert, audioconvert, audioresample,
357 Source elements such as audiotestsrc, videotestsrc, v4l2src,
363 Let's look at the example of an element that can convert between
364 samplerates, so where input and output samplerate don't have to be
370 gst_my_filter_setcaps (GstMyFilter *filter,
373 if (gst_pad_set_caps (filter->sinkpad, caps)) {
374 filter->passthrough = TRUE;
376 GstCaps *othercaps, *newcaps;
377 GstStructure *s = gst_caps_get_structure (caps, 0), *others;
379 /* no passthrough, setup internal conversion */
380 gst_structure_get_int (s, "channels", &filter->channels);
381 othercaps = gst_pad_get_allowed_caps (filter->srcpad);
382 others = gst_caps_get_structure (othercaps, 0);
383 gst_structure_set (others,
384 "channels", G_TYPE_INT, filter->channels, NULL);
386 /* now, the samplerate value can optionally have multiple values, so
387 * we "fixate" it, which means that one fixed value is chosen */
388 newcaps = gst_caps_copy_nth (othercaps, 0);
389 gst_caps_unref (othercaps);
390 gst_pad_fixate_caps (filter->srcpad, newcaps);
391 if (!gst_pad_set_caps (filter->srcpad, newcaps))
394 /* we are now set up, configure internally */
395 filter->passthrough = FALSE;
396 gst_structure_get_int (s, "rate", &filter->from_samplerate);
397 others = gst_caps_get_structure (newcaps, 0);
398 gst_structure_get_int (others, "rate", &filter->to_samplerate);
405 gst_my_filter_sink_event (GstPad *pad,
410 GstMyFilter *filter = GST_MY_FILTER (parent);
412 switch (GST_EVENT_TYPE (event)) {
417 gst_event_parse_caps (event, &caps);
418 ret = gst_my_filter_setcaps (filter, caps);
422 ret = gst_pad_event_default (pad, parent, event);
429 gst_my_filter_chain (GstPad *pad,
433 GstMyFilter *filter = GST_MY_FILTER (parent);
436 /* push on if in passthrough mode */
437 if (filter->passthrough)
438 return gst_pad_push (filter->srcpad, buf);
441 out = gst_my_filter_convert (filter, buf);
442 gst_buffer_unref (buf);
444 return gst_pad_push (filter->srcpad, out);
451 <sect1 id="section-nego-upstream" xreflabel="Upstream caps (re)negotiation">
452 <title>Upstream caps (re)negotiation</title>
454 Upstream negotiation's primary use is to renegotiate (part of) an
455 already-negotiated pipeline to a new format. Some practical examples
456 include to select a different video size because the size of the video
457 window changed, and the video output itself is not capable of rescaling,
458 or because the audio channel configuration changed.
461 Upstream caps renegotiation is requested by sending a GST_EVENT_RECONFIGURE
462 event upstream. The idea is that it will instruct the upstream element
463 to reconfigure its caps by doing a new query for the allowed caps and then
464 choosing a new caps. The element that sends out the RECONFIGURE event
465 would influence the selection of the new caps by returning the new
466 prefered caps from its GST_QUERY_CAPS query function. The RECONFIGURE
467 event will set the GST_PAD_FLAG_NEED_RECONFIGURE on all pads that it
471 It is important to note here that different elements actually have
472 different responsibilities here:
477 Elements that want to propose a new format upstream need to first
478 check if the new caps are acceptable upstream with an ACCEPT_CAPS
479 query. Then they would send a RECONFIGURE event and be prepared to
480 answer the CAPS query with the new prefered format. It should be
481 noted that when there is no upstream element that can (or wants)
482 to renegotiate, the element needs to deal with the currently
488 Elements that operate in transform negotiation according to
489 <xref linkend="section-nego-transform"/> pass the RECONFIGURE
490 event upstream. Because these elements simply do a fixed transform
491 based on the upstream caps, they need to send the event upstream
492 so that it can select a new format.
497 Elements that operate in fixed negotiation
498 (<xref linkend="section-nego-fixed"/>) drop the RECONFIGURE event.
499 These elements can't reconfigure and their output caps don't depend
500 on the upstream caps so the event can be dropped.
505 Elements that can be reconfigured on the source pad (source pads
506 implementing dynamic negotiation in
507 <xref linkend="section-nego-dynamic"/>) should check its
508 NEED_RECONFIGURE flag with
509 <function>gst_pad_check_reconfigure ()</function> and it should
510 start renegotiation when the function returns TRUE.
516 <sect1 id="section-nego-getcaps" xreflabel="Implementing a CAPS query function">
517 <title>Implementing a CAPS query function</title>
519 A <function>_query ()</function>-function with the GST_QUERY_CAPS query
520 type is called when a peer element would like to know which formats
521 this pad supports, and in what order of preference. The return value
522 should be all formats that this elements supports, taking into account
523 limitations of peer elements further downstream or upstream, sorted by
524 order of preference, highest preference first.
531 gst_my_filter_query (GstPad *pad, GstObject * parent, GstQuery * query)
534 GstMyFilter *filter = GST_MY_FILTER (parent);
536 switch (GST_QUERY_TYPE (query)) {
540 GstCaps *temp, *caps, *filt, *tcaps;
543 otherpad = (pad == filter->srcpad) ? filter->sinkpad :
545 caps = gst_pad_get_allowed_caps (otherpad);
547 gst_query_parse_caps (query, &filt);
549 /* We support *any* samplerate, indifferent from the samplerate
550 * supported by the linked elements on both sides. */
551 for (i = 0; i < gst_caps_get_size (caps); i++) {
552 GstStructure *structure = gst_caps_get_structure (caps, i);
554 gst_structure_remove_field (structure, "rate");
557 /* make sure we only return results that intersect our
559 tcaps = gst_pad_get_pad_template_caps (pad);
561 temp = gst_caps_intersect (caps, tcaps);
562 gst_caps_unref (caps);
563 gst_caps_unref (tcaps);
566 /* filter against the query filter when needed */
568 temp = gst_caps_intersect (caps, filt);
569 gst_caps_unref (caps);
572 gst_query_set_caps_result (query, caps);
573 gst_caps_unref (caps);
578 ret = gst_pad_query_default (pad, parent, query);
587 <sect1 id="section-nego-pullmode">
588 <title>Pull-mode Caps negotiation</title>
590 WRITEME, the mechanism of pull-mode negotiation is not yet fully
595 Using all the knowledge you've acquired by reading this chapter, you
596 should be able to write an element that does correct caps negotiation.
597 If in doubt, look at other elements of the same type in our git
598 repository to get an idea of how they do what you want to do.