Current Metadata handling
~~~~~~~~~~~~~~~~~~~~~~~~~
-When reading files, demuxers or parsers extract the metadata. It will be sent as
-a GST_MESSAGE_TAG to the GstBus and GST_EVENT_TAG to downstream elements.
+When reading files, demuxers or parsers extract the metadata. It will be sent
+a GST_EVENT_TAG to downstream elements. When a sink element receives a tag
+event, it will post a GST_MESSAGE_TAG message on the bus with the contents of
+the tag event.
Elements receiving GST_EVENT_TAG events can mangle them, mux them into the
buffers they send or just pass them through. Usually is muxers that will format
the tag data into the form required by the format they mux. Such elements would
also implement the GstTagSetter interface to receive tags from the application.
- GstMessage(tag) over GstBus to application
- ^
- |
+----------+
| demux |
sink src --> GstEvent(tag) over GstPad to downstream element
and another a push, the push might be busy while the block callback is done.
* maybe this name is overloaded. We need to look at some more use cases before
trying to fix this.
- FIXED in 0.11 with BLOCKING probes. Not everything is implemented yet, though.
+ FIXED in 0.11 with BLOCKING probes.
- rethink the way we do upstream renegotiation. Currently it's done with
pad_alloc but this has many issues such as only being able to suggest 1 format
This could, for example, easily be triggered with a RENEGOTIATE event.
FIXED in 0.11 with RECONFIGURE events.
-- Remove the result format value in queries.
+- Remove the result format value in queries. FIXED in 0.11
- Try to minimize the amount of acceptcaps calls when pushing buffers around.
The element pushing the buffer usually negotiated already and decided on the
will block when all buffers are outstanding until a buffer is returned to the
pool. This behaviour can be changed by specifying the
GST_BUFFER_POOL_FLAG_DONTWAIT flag in the parameters. With this flag set,
- allocation will return GST_FLOW_UNEXPECTED when the pool is empty.
+ allocation will return GST_FLOW_EOS when the pool is empty.
Renegotiation
~~~~~~~~~~
After the transform element is configured with caps, a bufferpool needs to be
-negotiated to perform the allocation of buffers. We habe 2 cases:
+negotiated to perform the allocation of buffers. We have 2 cases:
- The element is operating in passthrough we don't need to allocate a buffer
in the transform element.
src pads
--------
-A gst_pad_push_event() on a srcpad will first store the event in the sticky
-array before sending the event to the peer pad. If there is no peer pad, the
-gst_pad_push_event() function returns NOT_LINKED.
+A gst_pad_push_event() on a srcpad will first store the sticky event in the
+sticky array before sending the event to the peer pad. If there is no peer pad
+and the event was not stored in the sticky array, FALSE is returned.
-Note that the behaviour is not influenced by a flushing pad.
+Flushing pads will refuse the events and will not store the sticky events.
-FLUSH_START and FLUSH_STOP events are dropped on blocked pads.
sink pads
---------
-A gst_pad_send_event() on a sinkpad will check the new event against the
-existing event. If they are different, the new event is stored as a pending
-event. If the events are the same, nothing changes.
+A gst_pad_send_event() on a sinkpad will call the event function on the pad. If
+the event function returns success, the sticky event is stored in the sticky
+event array and the event is marked for update.
-When the pad is flushing, the _send_event() function returns WRONG_STATE
-immediately.
+When the pad is flushing, the _send_event() function returns FALSE immediately.
-The event function is then called for all pending events. If the function
-returns success, the pending event is copied to the active events, else the
-pending event is removed and the current active event is unchanged.
+When the next data item is pushed, the pending events are pushed first.
This ensures that the event function is never called for flushing pads and that
the sticky array only contains events for which the event function returned
pad link
--------
-When linking pads, all the sticky events from the srcpad are copied to the
-pending array on the sinkpad. The pending events will be sent to the event
-function of the sinkpad on the next event or buffer.
+When linking pads, the srcpad sticky events are marked for update when they are
+different from the sinkpad events. The next buffer push will push the events to
+the sinkpad.
FLUSH_START/STOP
source element when it has finished sending data. This event is mainly sent
in the streaming thread but can also be sent from the application thread.
-An EOS event sent on a srcpad returns GST_FLOW_UNEXPECTED.
+An EOS event sent on a srcpad returns GST_FLOW_EOS.
The downstream element should forward the EOS event to its downstream peer
elements. This way the event will eventually reach the sinks which should
Sometimes the EOS event is generated by another element than the source, for
example a demuxer element can generate an EOS event before the source element.
This is not a problem, the demuxer does not send an EOS event to the upstream
-element but returns GST_FLOW_UNEXPECTED, causing the source element to stop
+element but returns GST_FLOW_EOS, causing the source element to stop
sending data.
An element that sends EOS on a pad should stop sending data on that pad. Source
PAUSED->PLAYING in a non-live pipeline).
+Query
+~~~~~
+
+The pipeline latency is queried with the LATENCY query.
+
+ (out) "live", G_TYPE_BOOLEAN (default FALSE)
+ - if a live element is found upstream
+
+ (out) "min-latency", G_TYPE_UINT64 (default 0)
+ - the minimum latency in the pipeline
+
+ (out) "max-latency", G_TYPE_UINT64 (default 0)
+ - the maximum latency in the pipeline
+
+
+Event
+~~~~~
+
+The latency in the pipeline is configured with the LATENCY event, which contains
+the following fields:
+
+ "latency", G_TYPE_UINT64
+ - the configured latency in the pipeline
+
+
Latency compensation
~~~~~~~~~~~~~~~~~~~~
A GstMeta is a structure as follows:
struct _GstMeta {
+ GstMetaFlags flags;
const GstMetaInfo *info; /* tag and info for the meta item */
};
GstMiniObject | GType (GstBuffer) |
| refcount, flags, copy/disp/free |
+-------------------------------------+
-GstBuffer | caps, parent, pool |
+GstBuffer | pool,pts,dts,duration,offsets |
+ | <private data> |
+.....................................+
| next ---+
+- | info ------> GstMetaInfo
flexible, constrained by those parts of the pipeline that are not
flexible.
+
+Basic rules
+~~~~~~~~~~~
+
+The simple rules must be followed:
+
+ 1) downstream suggests formats
+ 2) upstream decides on format
+
+There are 4 queries/events used in caps negotiation:
+
+ 1) GST_QUERY_CAPS : get possible formats
+ 2) GST_QUERY_ACCEPT_CAPS : check if format is possible
+ 3) GST_EVENT_CAPS : configure format (downstream)
+ 4) GST_EVENT_RECONFIGURE : inform upstream of possibly new caps
+
+
+Queries
+-------
+
+A pad can ask the peer pad for its supported GstCaps. It does this with
+the CAPS query. The list of supported caps can be used to choose an
+appropriate GstCaps for the data transfer.
+
+ (in) "filter", GST_TYPE_CAPS (default NULL)
+ - a GstCaps to filter the results against
+
+ (out) "caps", GST_TYPE_CAPS (default NULL)
+ - the result caps
+
+
+A pad can ask the peer pad if it supports a given caps. It does this with
+the ACCEPT_CAPS query.
+
+ (in) "caps", GST_TYPE_CAPS
+ - a GstCaps to check
+
+ (out) "result", G_TYPE_BOOLEAN (default FALSE)
+ - TRUE if the caps are accepted
+
+
+Events
+~~~~~~
+
+When a media format is negotiated, peer elements are notified of the GstCaps
+with the CAPS event. The caps must be fixed.
+
+ "caps", GST_TYPE_CAPS
+ - the negotiated GstCaps
+
+
+Operation
+~~~~~~~~~
+
GStreamer's two scheduling modes, push mode and pull mode, lend
themselves to different mechanisms to achieve this goal. As it is more
common we describe push mode negotiation first.
+
Push-mode negotiation
~~~~~~~~~~~~~~~~~~~~~
- An element should reconfigure itself to the new format received as a CAPS
event before processing the following buffers. If the data type in the
- caps event is not acceptable, the element should refuse the buffer by
- returning an appropriate GST_FLOW_NOT_NEGOTIATED return value from the
- chain function.
+ caps event is not acceptable, the element should refuse the event. The
+ element should also refuse the next buffers by returning an appropriate
+ GST_FLOW_NOT_NEGOTIATED return value from the chain function.
- Downstream elements can request a format change of the stream by sending a
RECONFIGURE event upstream. Upstream elements will renegotiate a new format
src sink
| |
- | accepts? |
- type A |---------------->|
- | yes |
- |< - - - - - - - -|
- | |
+ | querycaps? |
+ |---------------->|
+ | caps |
+ select caps |< - - - - - - - -|
+ from the | |
+ candidates | |
+ | |-.
+ | accepts? | |
+ type A |---------------->| | optional
+ | yes | |
+ |< - - - - - - - -| |
+ | |-'
| send_event() |
send CAPS |---------------->| Receive type A, reconfigure to
event A | | process type A.
[element wants to create a buffer]
if not format
# see what we can do
- ourcaps = gst_pad_get_caps (srcpad)
+ ourcaps = gst_pad_query_caps (srcpad)
# see what the peer can do filtered against our caps
- candidates = gst_pad_peer_get_caps (srcpad, ourcaps)
+ candidates = gst_pad_peer_query_caps (srcpad, ourcaps)
foreach candidate in candidates
# make sure the caps is fixed
mark .-|<----------------| send RECONFIGURE event
renegotiate| | |
'>| |
- | get_caps() |
+ | querycaps() |
renegotiate |---------------->|
| suggest B |
|< - - - - - - - -|
1) Who decides what format to use?
- src pad always decides, by convention. sinkpad can suggest a format
- by putting it high in the getcaps function GstCaps.
+ by putting it high in the caps query result GstCaps.
- since the src decides, it can always choose something that it can do,
so this step can only fail if the sinkpad stated it could accept
something while later on it couldn't.
- src receives the RECONFIGURE event and marks renegotiation
- On the next buffer push, the source renegotiates the caps and the
bufferpool. The sink will put the new new prefered format high in the list
- of caps it returns from its getcaps function.
+ of caps it returns from its caps query.
videotestsrc ! queue ! xvimagesink
- - queue proxies all accept and getcaps to the other peer pad.
+ - queue proxies all accept and caps queries to the other peer pad.
- queue proxies the bufferpool
- queue proxies the RECONFIGURE event
- queue stores CAPS event in the queue. This means that the queue can contain
doing a SCHEDULING query.
The sink initiates the negotiation process by intersecting the results
-of gst_pad_get_caps() on its sink pad and its peer src pad. This is the
+of gst_pad_query_caps() on its sink pad and its peer src pad. This is the
operation performed by gst_pad_get_allowed_caps(). In the simple
-passthrough case, the peer pad's getcaps() function should return the
+passthrough case, the peer pad's caps query should return the
intersection of calling get_allowed_caps() on all of its sink pads. In
this way the sink element knows the capabilities of the entire pipeline.
The sink element then fixates the resulting caps, if necessary,
-resulting in the flow caps. From now on, the getcaps function
-of the sinkpad will only return these fixed caps meaning that upstream elements
+resulting in the flow caps. From now on, the caps query of the sinkpad
+will only return these fixed caps meaning that upstream elements
will only be able to produce this format.
If the sink element could not set caps on its sink pad, it should post
the sink. This is needed to provide a consistent API to the applications that
expect ASYNC return values from sinks but it also allows us to perform the
remainder of the negotiation outside of the context of the pulling thread.
+
+
+Patterns
+~~~~~~~~
+
+We can identify 3 patterns in negotiation:
+
+ 1) Fixed : Can't choose the output format
+ - Caps encoded in the stream
+ - A video/audio decoder
+ - usually uses gst_pad_use_fixed_caps()
+
+ 2) Passthrough
+ - Caps not modified
+ - can do caps transform based on element property
+ - videobox
+
+ 3) Dynamic : can choose output format
+ - A converter element
+ - depends on downstream caps, needs to do a CAPS query to find
+ transform.
+ - usually prefers to use the identity transform
This is essential to be able to implement dynamic relinking of elements
without breaking the dataflow.
- - be notified when data or events are pushed or sent on a pad. It should also
- be possible to inspect and modify the data.
+ - be notified when data, events or queries are pushed or sent on a pad. It
+ should also be possible to inspect and modify the data.
- be able to drop, pass and block on data based on the result of the callback.
GST_PAD_PROBE_TYPE_BLOCK = (1 << 1),
/* flags to select datatypes */
- GST_PAD_PROBE_TYPE_BUFFER = (1 << 2),
- GST_PAD_PROBE_TYPE_BUFFER_LIST = (1 << 3),
- GST_PAD_PROBE_TYPE_EVENT_DOWNSTREAM = (1 << 4),
- GST_PAD_PROBE_TYPE_EVENT_UPSTREAM = (1 << 5),
+ GST_PAD_PROBE_TYPE_BUFFER = (1 << 4),
+ GST_PAD_PROBE_TYPE_BUFFER_LIST = (1 << 5),
+ GST_PAD_PROBE_TYPE_EVENT_DOWNSTREAM = (1 << 6),
+ GST_PAD_PROBE_TYPE_EVENT_UPSTREAM = (1 << 7),
+ GST_PAD_PROBE_TYPE_EVENT_FLUSH = (1 << 8),
+ GST_PAD_PROBE_TYPE_QUERY_DOWNSTREAM = (1 << 9),
+ GST_PAD_PROBE_TYPE_QUERY_UPSTREAM = (1 << 10),
/* flags to select scheduling mode */
- GST_PAD_PROBE_TYPE_PUSH = (1 << 6),
- GST_PAD_PROBE_TYPE_PULL = (1 << 7),
+ GST_PAD_PROBE_TYPE_PUSH = (1 << 12),
+ GST_PAD_PROBE_TYPE_PULL = (1 << 13),
} GstPadProbeType;
the callback will be called as soon as the pad becomes idle in the streaming
thread.
- The IDLE probe in useful to perform dynamic linking, it allows to wait for for
+ The IDLE probe is useful to perform dynamic linking, it allows to wait for for
a safe moment when an unlink/link operation can be done. Since the probe is a
blocking probe, it will also make sure that the pad stays idle until the probe
is removed.
< - - - - - - O |
| |
+For queries, the PUSH ProbeType is set when the query is traveling to the object
+that will answer the query and the PULL type is set when the query contains the
+answer.
Query
-----
-NOTE: this is implemented as proposed
-
Purpose
~~~~~~~
Queries are used to get information about the stream.
A query is started on a specific pad and travels up or downstream.
-Types of queries
-~~~~~~~~~~~~~~~~
-
- - get length of stream
- - get position in stream
- - get seeking capabilities
- - get latency
- - convert one value to another
- - query supported formats
- - query internal links.
-
-Current implementation
-~~~~~~~~~~~~~~~~~~~~~~
-
- The current implementation of query requires pads to implement the
- following functions:
-
- gboolean (*GstPadConvertFunction) (GstPad *pad,
- GstFormat src_format, gint64 src_value,
- GstFormat *dest_format, gint64 *dest_value);
- gboolean (*GstPadQueryFunction) (GstPad *pad, GstQueryType type,
- GstFormat *format, gint64 *value);
- GList* (*GstPadIntLinkFunction) (GstPad *pad);
- const GstFormat* (*GstPadFormatsFunction) (GstPad *pad);
- const GstEventMask* (*GstPadEventMaskFunction) (GstPad *pad);
- const GstQueryType* (*GstPadQueryTypeFunction) (GstPad *pad);
-
- Most of these functions are not very extensible in particular,
- the queryfunction can only return one value.
-
Requirements
~~~~~~~~~~~~
- one pad function to perform the query
- extensible queries.
-Proposition
-~~~~~~~~~~~
+Implementation
+~~~~~~~~~~~~~~
- - define GstQuery extending GstMiniObject and containing a GstStructure (see GstMessage)
- - define standard query types (see proposed types)
- - define methods to create a parse the results in the GstQuery.
+ - GstQuery extends GstMiniObject and contains a GstStructure (see GstMessage)
+ - some standard query types are defined below
+ - methods to create and parse the results in the GstQuery.
- define pad method:
- gboolean (*GstPadQueryFunction) (GstPad *pad, GstQuery *query);
+ gboolean (*GstPadQueryFunction) (GstPad *pad,
+ GstObject *parent,
+ GstQuery *query);
pad returns result in query structure and TRUE as result or FALSE when
query is not supported.
-Proposed types
-~~~~~~~~~~~~~~
-
- - GST_QUERY_SEEKING:
-
- get info on how seeking can be done
- - getrange, with/without offset/size
- - ranges where seeking is efficient (for caching network sources)
- - flags describing seeking behaviour (forward, backward, segments,
- play backwards, ...)
+Query types
+~~~~~~~~~~~
- GST_QUERY_POSITION:
- GST_QUERY_LATENCY:
- get amount of latency introduced in the pipeline.
+ get amount of latency introduced in the pipeline. (See part-latency.txt)
+
+ - GST_QUERY_RATE:
+
+ get the current playback rate of the pipeline
+
+ - GST_QUERY_SEEKING:
+
+ get info on how seeking can be done
+ - getrange, with/without offset/size
+ - ranges where seeking is efficient (for caching network sources)
+ - flags describing seeking behaviour (forward, backward, segments,
+ play backwards, ...)
- GST_QUERY_SEGMENT:
return list of supported formats that can be used for GST_QUERY_CONVERT.
+ - GST_QUERY_BUFFERING:
+
+ query available media for efficient seeking (See part-buffering.txt)
+
+ - GST_QUERY_CUSTOM:
+
+ a custom query, the name of the query defines the properties of the query.
+
+ - GST_QUERY_URI:
+
+ query the uri of the source or sink element
+
+ - GST_QUERY_ALLOCATION:
+
+ the buffer allocation properties (See part-bufferpool.txt)
+
+ - GST_QUERY_SCHEDULING:
+
+ the scheduling properties (See part-scheduling.txt)
+
+ - GST_QUERY_ACCEPT_CAPS:
+
+ check if caps are supported (See part-negotiation.txt)
+
+ - GST_QUERY_CAPS:
+
+ get the possible caps (See part-negotiation.txt)
+
+
segment, the decoder knows it should not send the P frame.
Avidemux stops sending data after pushing a frame with timestamp 5 and
- returns GST_FLOW_UNEXPECTED from the chain function to make the upstream
+ returns GST_FLOW_EOS from the chain function to make the upstream
elements perform the EOS logic.
3) EOS
- marks the end of data, nothing is to be expected after EOS, elements
- should refuse more data and return GST_FLOW_UNEXPECTED. A FLUSH_STOP
+ should refuse more data and return GST_FLOW_EOS. A FLUSH_STOP
event clears the EOS state of an element.
* gst_query_set_latency:
* @query: a #GstQuery
* @live: if there is a live element upstream
- * @min_latency: the minimal latency of the live element
- * @max_latency: the maximal latency of the live element
+ * @min_latency: the minimal latency of the upstream elements
+ * @max_latency: the maximal latency of the upstream elements
*
* Answer a latency query by setting the requested values in the given format.
*
projects / platform / upstream / gstreamer.git / commitdiff