4 This document outlines the details of the frame stepping functionality in
7 The stepping functionality operates on the current playback segment, position
8 and rate as it was configured with a regular seek event. In contrast to the seek
9 event, it operates very closely to the sink and thus has a very low latency and
10 is not slowed down by queues and does not actually perform any seeking logic.
11 For this reason we want to include a new API instead of reusing the seek API.
13 The following requirements are needed:
15 - The ability to walk forwards and backwards in the stream.
16 - Arbitrary increments in any supported format (time, frames, bytes ...)
17 - High speed, minimal overhead. This mechanism is not more expensive than
19 - switching between forwards and backwards stepping should be fast.
20 - Maintain synchronisation between streams.
21 - Get feedback of the amount of skipped data.
22 - Ability to play a certain amount of data at an arbitrary speed.
24 We want a system where we can step frames in PAUSED as well as play short
25 segments of data in PLAYING.
31 * frame stepping in video only pipeline in PAUSED
33 .-----. .-------. .------. .-------.
34 | src | | demux | .-----. | vdec | | vsink |
35 | src->sink src1->|queue|->sink src->sink |
36 '-----' '-------' '-----' '------' '-------'
38 - app sets the pipeline to PAUSED to block on the preroll picture
39 - app seeks to required position in the stream. This can be done with a
40 positive or negative rate depending on the required frame stepping
42 - app steps frames (in GST_FORMAT_DEFAULT or GST_FORMAT_BUFFER). The
43 pipeline loses its PAUSED state until the required number of frames have
44 been skipped, it then prerolls again. This skipping is purely done in
46 - sink posts STEP_DONE with amount of frames stepped and corresponding time
49 * frame stepping in audio/video pipeline in PAUSED
51 .-----. .-------. .------. .-------.
52 | src | | demux | .-----. | vdec | | vsink |
53 | src->sink src1->|queue|->sink src->sink |
54 '-----' | | '-----' '------' '-------'
55 | | .------. .-------.
56 | | .-----. | adec | | asink |
57 | src2->|queue|->sink src->sink |
58 '-------' '-----' '------' '-------'
61 - app sets the pipeline to PAUSED to block on the preroll picture
62 - app seeks to required position in the stream. This can be done with a
63 positive or negative rate depending on the required frame stepping
65 - app steps frames (in GST_FORMAT_DEFAULT or GST_FORMAT_BUFFER) or an amount
66 of time on the video sink. The pipeline loses its PAUSED state until the
67 required number of frames have been skipped, it then prerolls again.
68 This skipping is purely done in the sink.
69 - sink posts STEP_DONE with amount of frames stepped and corresponding time
71 - the app skips the same amount of time on the audiosink to align the
72 streams again. When huge amount of video frames are skipped, there needs
73 to be enough queueing in the pipeline to compensate for the accumulated
76 * frame stepping in audio/video pipeline in PLAYING
78 - app sets the pipeline to PAUSED to block on the preroll picture
79 - app seeks to required position in the stream. This can be done with a
80 positive or negative rate depending on the required frame stepping
82 - app configures frames steps (in GST_FORMAT_DEFAULT or GST_FORMAT_BUFFER) or
83 an amount of time on the sink. The step event has a flag indicating live
84 stepping so that the stepping will only happens in PLAYING.
85 - app sets pipeline to PLAYING. The pipeline continues PLAYING until it
86 consumed the amount of time.
87 - sink posts STEP_DONE with amount of frames stepped and corresponding time
88 interval. The sink will then wait for another step event. Since the
89 STEP_DONE message was emitted by the sink when it handed off the buffer to
90 the device, there is usually sufficient time to queue a new STEP event so
91 that one can seamlessly continue stepping.
97 A new GST_EVENT_STEP event is introduced to start the step operation.
98 The step event is created with the following fields in the structure:
100 "format", GST_TYPE_FORMAT
101 The format of the step units
103 "amount", G_TYPE_UINT64
104 The amount of units to step. A 0 amount immediately completes and can be
105 used to cancel the current step and resume normal non-stepping behaviour
106 to the end of the segment.
107 A -1 amount steps until the end of the segment.
109 "rate", G_TYPE_DOUBLE
110 The rate at which the frames should be stepped in PLAYING mode. 1.0 is
111 the normal playback speed and direction of the segment, 2.0
112 is double speed. A speed of 0.0 is not allowed. When performing a
113 flushing step, the speed is not relevant. Note that we don't allow negative
114 rates here, use a seek with a negative rate first to reverse the playback
117 "flush", G_TYPE_BOOLEAN
118 when flushing is TRUE, the step is performed immediately:
120 - In the PAUSED state the pipeline loses the PAUSED state, the requested
121 amount of data is skipped and the pipeline prerolls again when a
122 non-intermediate step completes.
123 When the pipeline was stepping while the event is sent, the current step
124 operation is updated with the new amount and format. The sink will do a
125 best effort to comply with the new amount.
126 - In the PLAYING state, the pipeline loses the PLAYING state, the
127 requested amount of data is skipped (not rendered) from the previous STEP
128 request or from the position of the last PAUSED if no previous STEP
129 operation was performed. The pipeline goes back to the PLAYING state
130 when a non-intermediate step completes.
132 When flushing is FALSE, the step will be performed later.
134 - In the PAUSED state the step will be done when going to PLAYING. Any
135 previous step operation will be overridden with the new STEP event.
136 - In the PLAYING state the step operation will be performed after the
137 current step operation completes. If there was no previous step
138 operation, the step operation will be performed from the position of the
141 "intermediate", G_TYPE_BOOLEAN
142 Signal that this step operation is an intermediate step, part of a series
143 of step operations. It is mostly interesting for stepping in the PAUSED state
144 because the sink will only perform a preroll after a non-intermediate step
145 operation completes. Intermediate steps are useful to flush out data from
146 other sinks in order to not cause excessive queueing. In the PLAYING state
147 the intermediate flag has no visual effect. In all states, the intermediate
148 flag is passed to the corresponding GST_MESSAGE_STEP_DONE.
151 The application will create a STEP event to start or stop the stepping
152 operation. Both stepping in PAUSED and PLAYING can be performed by means of
155 The event is usually sent to the pipeline, which will typically distribute the
156 event to all of its sinks. For some use cases, like frame stepping on video
157 frames only, the event should only be sent to the video sink and upon reception
158 of the STEP_DONE message, one can step the other sinks to align the streams
161 For large stepping amounts, there needs to be enough queueing in front of all
162 the sinks. If large steps need to be performed, they can be split up into
163 smaller step operations using the "intermediate" flag on the step.
165 Since the step event does not update the base_time of any of the elements, the
166 sinks should keep track of the amount of stepped data in order to remain
167 synchronized against the clock.
173 A GST_MESSAGE_STEP_START is created. It contains the following fields.
176 If the step was queued or activated.
178 "format", GST_TYPE_FORMAT
179 The format of the step units that queued/activated.
181 "amount", G_TYPE_UINT64
182 The amount of units that were queued/activated.
184 "rate", G_TYPE_DOUBLE
185 The rate and direction at which the frames were queued/activated.
187 "flush", G_TYPE_BOOLEAN
188 If the queued/activated frames will be flushed.
190 "intermediate", G_TYPE_BOOLEAN
191 If this is an intermediate step operation that queued/activated.
193 The STEP_START message is emitted 2 times:
195 * first when an element received the STEP event and queued it. The "active"
196 field will be FALSE in this case.
198 * second when the step operation started in the streaming thread. The "active"
199 field is TRUE in this case. After this message is emitted, the application
200 can queue a new step operation.
202 The purpose of this message is to find out how many elements participate in the
203 step operation and to queue new step operations at the earliest possible
206 A new GST_MESSAGE_STEP_DONE message is created. It contains the following
209 "format", GST_TYPE_FORMAT
210 The format of the step units that completed.
212 "amount", G_TYPE_UINT64
213 The amount of units that were stepped.
215 "rate", G_TYPE_DOUBLE
216 The rate and direction at which the frames were stepped.
218 "flush", G_TYPE_BOOLEAN
219 If the stepped frames were flushed.
221 "intermediate", G_TYPE_BOOLEAN
222 If this is an intermediate step operation that completed.
224 "duration", G_TYPE_UINT64
225 The total duration of the stepped units in GST_FORMAT_TIME.
227 "eos", G_TYPE_BOOLEAN
228 The step ended because of EOS.
230 The message is emitted by the element that performs the step operation. The
231 purpose is to return the duration in GST_FORMAT_TIME of the stepped media. This
232 especially interesting to align other stream in case of stepping frames on the
239 When quickly switching between a forwards and a backwards step of, for example,
240 one video frame, we need either:
242 a) issue a new seek to change the direction from the current position.
243 b) cache a certain number of stepped frames and walk the cache.
245 option a) might be very slow.
246 For option b) we would ideally like to offload this caching functionality to a
247 separate element, which means that we need to forward the STEP event upstream.
248 It's unclear how this could work in a generic way. What is a demuxer supposed
249 to do when it received a step event? a flushing seek to what stream position?