From a13be0a71e6cff4aab591f3f7ebdb3628f374f2a Mon Sep 17 00:00:00 2001 From: "Ronald S. Bultje" Date: Wed, 6 Jul 2005 12:18:00 +0000 Subject: [PATCH] Add a chapter on caps negotiation, simplify the original code samples a bit w.r.t. caps negotiation, add link to the ... Original commit message from CVS: * docs/pwg/advanced-negotiation.xml: * docs/pwg/building-boiler.xml: * docs/pwg/building-pads.xml: * docs/pwg/pwg.xml: * examples/pwg/Makefile.am: Add a chapter on caps negotiation, simplify the original code samples a bit w.r.t. caps negotiation, add link to the advanced section. Add a bunch of examples showing different use cases of different types of caps negotiation. Upstream renegotiation isn't fully documented yet since nobody knows how that works. --- ChangeLog | 13 ++ docs/pwg/advanced-negotiation.xml | 439 +++++++++++++++++++++++++++++++++++++ docs/pwg/building-boiler.xml | 2 + docs/pwg/building-pads.xml | 131 ++++------- docs/pwg/pwg.xml | 2 + examples/pwg/Makefile.am | 13 +- tests/old/examples/pwg/Makefile.am | 13 +- 7 files changed, 526 insertions(+), 87 deletions(-) create mode 100644 docs/pwg/advanced-negotiation.xml diff --git a/ChangeLog b/ChangeLog index b11abfe..037b375 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,3 +1,16 @@ +2005-07-06 Ronald S. Bultje + + * docs/pwg/advanced-negotiation.xml: + * docs/pwg/building-boiler.xml: + * docs/pwg/building-pads.xml: + * docs/pwg/pwg.xml: + * examples/pwg/Makefile.am: + Add a chapter on caps negotiation, simplify the original code + samples a bit w.r.t. caps negotiation, add link to the advanced + section. Add a bunch of examples showing different use cases of + different types of caps negotiation. Upstream renegotiation isn't + fully documented yet since nobody knows how that works. + 2005-07-06 Thomas Vander Stichele * check/gst/gstpad.c: diff --git a/docs/pwg/advanced-negotiation.xml b/docs/pwg/advanced-negotiation.xml new file mode 100644 index 0000000..b5a71e1 --- /dev/null +++ b/docs/pwg/advanced-negotiation.xml @@ -0,0 +1,439 @@ + + Caps negotiation + + Caps negotiation is the process where elements configure themselves + and each other for streaming a particular media format over their pads. + Since different types of elements have different requirements for the + media formats they can negotiate to, it is important that this process + is generic and implements all those use cases correctly. + + + In this chapter, we will discuss downstream negotiation and upstream + negotiation from a pipeline perspective, implicating the responsibilities + of different types of elements in a pipeline, and we will introduce the + concept of fixed caps. + + + + Caps negotiation use cases + + Let's take the case of a file source, linked to a demuxer, linked to a + decoder, linked to a converter with a caps filter and finally an audio + output. When dataflow originally starts, the demuxer will parse the + file header (e.g. the Ogg headers), and notice that there is, for + example, a Vorbis stream in this Ogg file. Noticing that, it will + create an output pad for the Vorbis elementary stream and set a + Vorbis-caps on it. Lastly, it adds the pad. As of this point, the pad + is ready to be used to stream data, and so the Ogg demuxer is now done. + This pad is not re-negotiatable, since the type of + the data stream is embedded within the data. + + + The Vorbis decoder will decode the Vorbis headers and the Vorbis data + coming in on its sinkpad. Now, some decoders may be able to output in + multiple output formats, for example both 16-bit integer output and + floating-point output, whereas other decoders may be able to only decode + into one specific format, e.g. only floating-point (32-bit) audio. Those + two cases have consequences for how caps negotiation should be + implemented in this decoder element. In the one case, it is possible to + use fixed caps, and you're done. In the other case, however, you should + implement the possibility for renegotiation in this + element, which is the possibility for the data format to be changed to + another format at some point in the future. We will discuss how to do + this in one of the sections further on in this chapter. + + + The filter can be used by applications to force, for example, a specific + channel configuration (5.1/surround or 2.0/stereo), on the pipeline, so + that the user can enjoy sound coming from all its speakers. The audio + sink, in this example, is a standard ALSA output element (alsasink). + The converter element supports any-to-any, and the filter will make sure + that only a specifically wanted channel configuration streams through + this link (as provided by the user's channel configuration preference). + By changing this preference while the pipeline is running, some elements + will have to renegotiate while the pipeline is + running. This is done through upstream caps renegotiation. + That, too, will be discussed in detail in a section further below. + + + In order for caps negotiation on non-fixed links to work correctly, + pads can optionally implement a function that tells peer elements what + formats it supports and/or preferes. When upstream renegotiation is + triggered, this becomes important. + + + Downstream elements are notified of a newly set caps only when data + is actually passing their pad. This is because caps is attached to + buffers during dataflow. So when the vorbis decoder sets a caps on + its source pad (to configure the output format), the converter will + not yet be notified. Instead, the converter will only be notified + when the decoder pushes a buffer over its source pad to the converter. + Right before calling the chain-function in the converter, &GStreamer; + will check whether the format that was previously negotiated still + applies to this buffer. If not, it first calls the setcaps-function + of the converter to configure it for the new format. Only after that + will it call the chain function of the converter. + + + + + Fixed caps + + The simplest way in which to do caps negotiation is setting a fixed + caps on a pad. After a fixed caps has been set, the pad can not be + renegotiated from the outside. The only way to reconfigure the pad + is for the element owning the pad to set a new fixed caps on the pad. + Fixed caps is a setup property for pads, called when creating the pad: + + +[..] + pad = gst_pad_new_from_template (..); + gst_pad_use_fixed_caps (pad); +[..] + + + The fixed caps can then be set on the pad by calling + gst_pad_set_caps (). + + +[..] + caps = gst_caps_new_simple ("audio/x-raw-float", + "width", G_TYPE_INT, 32, + "endianness", G_TYPE_INT, G_BYTE_ORDER, + "buffer-frames", G_TYPE_INT, <bytes-per-frame>, + "rate", G_TYPE_INT, <samplerate>, + "channels", G_TYPE_INT, <num-channels>, NULL); + if (!gst_pad_set_caps (pad, caps)) { + GST_ELEMENT_ERROR (element, CORE, NEGOTIATION, (NULL), + ("Some debug information here")); + return GST_FLOW_ERROR; + } +[..] + + + Elements that could implement fixed caps (on their source pads) are, + in general, all elements that are not renegotiatable. Examples include: + + + + + A typefinder, since the type found is part of the actual data stream + and can thus not be re-negotiated. + + + + + Pretty much all demuxers, since the contained elementary data + streams are defined in the file headers, and thus not + renegotiatable. + + + + + Some decoders, where the format is embedded in the datastream + and not part of the peercaps and where the + decoder itself is not reconfigureable, too. + + + + + All other elements that need to be configured for the format should + implement full caps negotiation, which will be explained in the next + few sections. + + + + + Downstream caps negotiation + + Downstream negotiation takes place when a format needs to be set on a + source pad to configure the output format, but this element allows + renegotiation because its format is configured on the sinkpad caps, + or because it supports multiple formats. The requirements for doing + the actual negotiation differ slightly. + + + + Negotiating caps embedded in input caps + + Many elements, particularly effects and converters, will be able + to parse the format of the stream from their input caps, and decide + the output format right at that time already. When renegotiation + takes place, some may merely need to "forward" the renegotiation + backwards upstream (more on that later). For those elements, all + (downstream) caps negotiation can be done in something that we + call the _setcaps () function. This function is + called when a buffer is pushed over a pad, but the format on this + buffer is not the same as the format that was previously negotiated + (or, similarly, no format was negotiated yet so far). + + + In the _setcaps ()-function, the element can + forward the caps to the next element and, if that pad accepts the + format too, the element can parse the relevant parameters from the + caps and configure itself internally. The caps passed to this function + is always a subset of the template caps, so + there's no need for extensive safety checking. The following example + should give a clear indication of how such a function can be + implemented: + + + +static gboolean +gst_my_filter_setcaps (GstPad *pad, + GstCaps *caps) +{ + GstMyFilter *filter = GST_MY_FILTER (GST_OBJECT_PARENT (pad)); + GstStructure *s; + + /* forward-negotiate */ + if (!gst_pad_set_caps (filter->srcpad, caps)) + return FALSE; + + /* negotiation succeeded, so now configure ourselves */ + s = gst_caps_get_structure (caps, 0); + gst_structure_get_int (s, "rate", &filter->samplerate); + gst_structure_get_int (s, "channels", &filter->channels); + + return TRUE; +} + + + + There may also be cases where the filter actually is able to + change the format of the stream. In those cases, + it will negotiate a new format. Obviously, the element should first + attempt to configure pass-through, which means that + it does not change the stream's format. However, if that fails, + then it should call gst_pad_get_allowed_caps () + on its sourcepad to get a list of supported formats on the outputs, + and pick the first. The return value of that function is guaranteed + to be a subset of the template caps. + + + Let's look at the example of an element that can convert between + samplerates, so where input and output samplerate don't have to be + the same: + + + +static gboolean +gst_my_filter_setcaps (GstPad *pad, + GstCaps *caps) +{ + GstMyFilter *filter = GST_MY_FILTER (GST_OBJECT_PARENT (pad)); + + if (gst_pad_set_caps (filter->sinkpad, caps)) { + filter->passthrough = TRUE; + } else { + GstCaps *othercaps, *newcaps; + GstStructure *s = gst_caps_get_structure (caps, 0), *others; + + /* no passthrough, setup internal conversion */ + gst_structure_get_int (s, "channels", &filter->channels); + othercaps = gst_pad_get_allowed_caps (filter->srcpad); + others = gst_caps_get_structure (othercaps, 0); + gst_structure_set (others, + "channels", G_TYPE_INT, filter->channels, NULL); + + /* now, the samplerate value can optionally have multiple values, so + * we "fixate" it, which means that one fixed value is chosen */ + newcaps = gst_caps_copy_nth (othercaps, 0); + gst_caps_unref (othercaps); + gst_pad_fixate_caps (filter->srcpad, newcaps); + if (!gst_pad_set_caps (filter->srcpad, newcaps)) + return FALSE; + + /* we are now set up, configure internally */ + filter->passthrough = FALSE; + gst_structure_get_int (s, "rate", &filter->from_samplerate); + others = gst_caps_get_structure (newcaps, 0); + gst_structure_get_int (others, "rate", &filter->to_samplerate); + } + + return TRUE; +} + +static GstFlowReturn +gst_my_filter_chain (GstPad *pad, + GstBuffer *buf) +{ + GstMyFilter *filter = GST_MY_FILTER (GST_OBJECT_PARENT (pad)); + GstBuffer *out; + + /* push on if in passthrough mode */ + if (filter->passthrough) + return gst_pad_push (filter->srcpad, buf); + + /* convert, push */ + out = gst_my_filter_convert (filter, buf); + gst_buffer_unref (buf); + + return gst_pad_push (filter->srcpad, out); +} + + + + + + Parsing and setting caps + + Other elements, such as certain types of decoders, will not be able + to parse the caps from their input, simply because the input format + does not contain the information required to know the output format + yet; rather, the data headers need to be parsed, too. In many cases, + fixed-caps will be enough, but in some cases, particularly in cases + where such decoders are renegotiatable, it is also possible to use + full caps negotiation. + + + Fortunately, the code required to do so is very similar to the last + code example in , with + the difference being that the caps is selected in the _chain + ()-function rather than in the _setcaps + ()-function. The rest, as for getting all allowed caps from + the source pad, fixating and such, is all the same. Re-negotiation, + which will be handled in the next section, is very different for such + elements, though. + + + + + + Upstream caps (re)negotiation + + Upstream negotiation's primary use is to renegotiate (part of) an + already-negotiated pipeline to a new format. Some practical examples + include to select a different video size because the size of the video + window changed, and the video output itself is not capable of rescaling, + or because the audio channel configuration changed. + + + Upstream caps renegotiation is done in the gst_pad_alloc_buffer + ()-function. The idea here is that an element requesting a + buffer from downstream, has to specify the type of that buffer. If + renegotiation is to take place, this type will no longer apply, and the + downstream element will set a new caps on the provided buffer. Next, + &GStreamer; will trigger renegotiation on the sourcepad of the element + before the function returns. + + + It is important to note here that different elements actually have + different responsibilities here: + + + + + Elements should implement a padalloc-function in + order to be able to change format on renegotiation. This is also + true for filters and converters. + + + + + Elements should allocate new buffers using + gst_pad_alloc_buffer (). + + + + + Elements that are renegotiatable should implement a + setcaps-function on their sourcepad as well. + + + + + Unfortunately, not all details here have been worked out yet, so this + documentation is incomplete. FIXME. + + + + + Implementing a getcaps function + + A _getcaps ()-function is called when a peer + element would like to know which formats this element supports, and + in what order of preference. The return value should be all formats + that this elements supports, taking into account limitations of peer + elements further downstream or upstream, sorted by order of preference, + highest preference first. + + + + + +static GstCaps * +gst_my_filter_getcaps (GstPad *pad) +{ + GstMyFilter *filter = GST_MY_FILTER (GST_OBJECT_PARENT (pad)); + GstPad *otherpad = (pad == filter->srcpad) ? filter->sinkpad : + filter->srcpad; + GstCaps *othercaps = gst_pad_get_allowed_caps (otherpad), *caps; + gint i; + + /* We support *any* samplerate, indifferent from the samplerate + * supported by the linked elements on both sides. */ + for (i = 0; i < gst_caps_get_size (othercaps); i++) { + GstStructure *structure = gst_caps_get_structure (othercaps, i); + + gst_structure_remove_field (structure, "rate"); + } + caps = gst_caps_intersect (othercaps, gst_pad_get_pad_template_caps (pad)); + gst_caps_unref (othercaps); + + return caps; +} + + + + Using all the knowledge you've acquired by reading this chapter, you + should be able to write an element that does correct caps negotiation. + If in doubt, look at other elements of the same type in our CVS + repository to get an idea of how they do what you want to do. + + + diff --git a/docs/pwg/building-boiler.xml b/docs/pwg/building-boiler.xml index a4e74af..a38ceef 100644 --- a/docs/pwg/building-boiler.xml +++ b/docs/pwg/building-boiler.xml @@ -147,6 +147,8 @@ typedef struct _GstMyFilter { } GstMyFilter; diff --git a/docs/pwg/building-pads.xml b/docs/pwg/building-pads.xml index f680085..ab445e4 100644 --- a/docs/pwg/building-pads.xml +++ b/docs/pwg/building-pads.xml @@ -8,23 +8,20 @@ of your element, and that makes them a very important item in the process of element creation. In the boilerplate code, we have seen how static pad templates take care of registering pad templates with the element class. - Here, we will see how to create actual elements, use _link () - and _getcaps () functions to let other elements know - their capabilities and how to register functions to let data flow through - the element. + Here, we will see how to create actual elements, use a _setcaps + ()-functions to configure for a particular format and how to + register functions to let data flow through the element. In the element _init () function, you create the pad from the pad template that has been registered with the element class in the _base_init () function. After creating the pad, - you have to set a _link () function pointer and a - _getcaps () function pointer. Optionally, you can - set a _chain () function pointer (on sink pads in - filter and sink elements) through which data will come in to the element, - or (on source pads in source elements) a _get () - function pointer through which data will be pulled from the element. After - that, you have to register the pad with the element. This happens like - this: + you have to set a _setcaps () function pointer and + optionally a _getcaps () function pointer. Also, you + have to set a _chain () function pointer. + Alternatively, pads can also operate in looping mode, which mans that they + can pull data themselves. More on this topic later. After that, you have + to register the pad with the element. This happens like this: static gboolean gst_my_filter_setcaps (GstPad *pad, GstCaps *caps); -static GstCaps * gst_my_filter_getcaps (GstPad *pad); static GstFlowReturn gst_my_filter_chain (GstPad *pad, GstBuffer *buf); @@ -95,10 +92,10 @@ gst_my_filter_init (GstMyFilter *filter) filter->sinkpad = gst_pad_new_from_template ( gst_element_class_get_pad_template (klass, "sink"), "sink"); gst_pad_set_setcaps_function (filter->sinkpad, gst_my_filter_setcaps); - gst_pad_set_getcaps_function (filter->sinkpad, gst_my_filter_getcaps); gst_pad_set_chain_function (filter->sinkpad, gst_my_filter_chain); @@ -107,30 +104,33 @@ gst_my_filter_init (GstMyFilter *filter) /* pad through which data goes out of the element */ filter->srcpad = gst_pad_new_from_template ( gst_element_class_get_pad_template (klass, "src"), "src"); - gst_pad_set_setcaps_function (filter->srcpad, gst_my_filter_setcaps); + + + gst_element_add_pad (GST_ELEMENT (filter), filter->srcpad); /* properties initial value */ filter->silent = FALSE; } - + - The link function + The setcaps-function - The _link () is called during caps negotiation. This - is the process where the linked pads decide on the streamtype that will - transfer between them. A full list of type-definitions can be found in - . A _link () + The _setcaps ()-function is called during caps + negotiation, which is discussed in great detail in . This is the process where the linked + pads decide on the streamtype that will transfer between them. A full + list of type-definitions can be found in . A _link () receives a pointer to a GstCaps - struct that defines the proposed streamtype, and can respond with - either yes (GST_PAD_LINK_OK), - no (GST_PAD_LINK_REFUSED) or - don't know yet (GST_PAD_LINK_DELAYED). - If the element responds positively towards the streamtype, that type - will be used on the pad. An example: + url="../../gstreamer/html/gstreamer-GstCaps.html">GstCaps + struct that defines the proposed streamtype, and can respond with + either yes (TRUE) or no + (FALSE). If the element responds positively towards + the streamtype, that type will be used on the pad. An example: static gboolean @@ -138,10 +138,7 @@ gst_my_filter_setcaps (GstPad *pad, GstCaps *caps) { GstStructure *structure = gst_caps_get_structure (caps, 0); - GstMyFilter *filter = GST_MY_FILTER (gst_pad_get_parent (pad)); - GstPad *otherpad = (pad == filter->srcpad) ? filter->sinkpad : - filter->srcpad; - GstPadLinkReturn ret; + GstMyFilter *filter = GST_MY_FILTER (GST_OBJECT_PARENT (pad)); const gchar *mime; /* Since we're an audio filter, we want to handle raw audio @@ -157,9 +154,8 @@ gst_my_filter_setcaps (GstPad *pad, /* we're a filter and don't touch the properties of the data. * That means we can set the given caps unmodified on the next * element, and use that negotiation return value as ours. */ - ret = gst_pad_set_caps (otherpad, gst_caps_copy (caps)); - if (GST_PAD_LINK_FAILED (ret)) - return ret; + if (!gst_pad_set_caps (filter->srcpad, caps)) + return FALSE; /* Capsnego succeeded, get the stream properties for internal * usage and return success. */ @@ -169,9 +165,21 @@ gst_my_filter_setcaps (GstPad *pad, g_print ("Caps negotiation succeeded with %d Hz @ %d channels\n", filter->samplerate, filter->channels); - return ret; + return TRUE; +} + + + --> In here, we check the mimetype of the provided caps. Normally, you don't need to do that in your own plugin/element, because the core does that @@ -188,58 +196,11 @@ gst_my_filter_setcaps (GstPad *pad, If your _link () function does not need to perform any specific operation (i.e. it will only forward caps), you can set it - to gst_pad_proxy_link. This is a link forwarding + to gst_pad_proxy_link (). This is a link forwarding function implementation provided by the core. It is useful for elements such as identity. - - - The getcaps function - - The _getcaps () function is used to request the list - of supported formats and properties from the element. In some cases, this - will be equal to the formats provided by the pad template, in which case - this function can be omitted. In some cases, too, it will not depend on - anything inside this element, but it will rather depend on the input from - another element linked to this element's sink or source pads. In that case, - you can use gst_pad_proxy_getcaps as implementation, - it provides getcaps forwarding in the core. However, in many cases, the - format supported by this element cannot be defined externally, but is - more specific than those provided by the pad template. In this case, you - should use a _getcaps () function. In the case as - specified below, we assume that our filter is able to resample sound, so - it would be able to provide any samplerate (indifferent from the samplerate - specified on the other pad) on both pads. It explains how a - _getcaps () can be used to do this. - - -static GstCaps * -gst_my_filter_getcaps (GstPad *pad) -{ - GstMyFilter *filter = GST_MY_FILTER (gst_pad_get_parent (pad)); - GstPad *otherpad = (pad == filter->srcpad) ? filter->sinkpad : - filter->srcpad; - GstCaps *othercaps = gst_pad_get_allowed_caps (otherpad), *caps; - gint i; - - if (gst_caps_is_empty (othercaps)) - return othercaps; - - /* We support *any* samplerate, indifferent from the samplerate - * supported by the linked elements on both sides. */ - for (i = 0; i < gst_caps_get_size (othercaps); i++) { - GstStructure *structure = gst_caps_get_structure (othercaps, i); - - gst_structure_remove_field (structure, "rate"); - } - caps = gst_caps_intersect (othercaps, gst_pad_get_pad_template_caps (pad)); - gst_caps_unref (othercaps); - - return caps; -} - -