+Support for Planar (Non-Interleaved) Raw Audio
+
+Raw audio samples are usually passed around in interleaved form in
+GStreamer, which means that if there are multiple audio channels the
+samples for each channel are interleaved in memory, e.g.
+|LEFT|RIGHT|LEFT|RIGHT|LEFT|RIGHT| for stereo audio. A non-interleaved
+or planar arrangement in memory would look like
+|LEFT|LEFT|LEFT|RIGHT|RIGHT|RIGHT| instead, possibly with
+|LEFT|LEFT|LEFT| and |RIGHT|RIGHT|RIGHT| residing in separate memory
+chunks or separated by some padding.
+
+GStreamer has always had signalling for non-interleaved audio since
+version 1.0, but it was never actually properly implemented in any
+elements. audioconvert would advertise support for it, but wasn’t
+actually able to handle it correctly.
+
+With this release we now have full support for non-interleaved audio as
+well, which means more efficient integration with external APIs that
+handle audio this way, but also more efficient processing of certain
+operations like interleaving multiple 1-channel streams into a
+multi-channel stream which can be done without memory copies now.
+
+New API to support this has been added to the GStreamer Audio support
+library: There is now a new GstAudioMeta which describes how data is
+laid out inside the buffer, and buffers with non-interleaved audio must
+always carry this meta. To access the non-interleaved audio samples you
+must map such buffers with gst_audio_buffer_map() which works much like
+gst_buffer_map() or gst_video_frame_map() in that it will populate a
+little GstAudioBuffer helper structure passed to it with the number of
+samples, the number of planes and pointers to the start of each plane in
+memory. This function can also be used to map interleaved audio buffers
+in which case there will be only one plane of interleaved samples.
+
+Of course support for this has also been implemented in the various
+audio helper and conversion APIs, base classes, and in elements such as
+audioconvert, audioresample, audiotestsrc, audiorate.
+
+Support for Closed Captions and Other Ancillary Data in Video
+
+The video support library has gained support for detecting and
+extracting Ancillary Data from videos as per the SMPTE S291M
+specification, including:
+
+- a VBI (Vertical Blanking Interval) parser that can detect and
+ extract Ancillary Data from Vertical Blanking Interval lines of
+ component signals. This is currently supported for videos in v210
+ and UYVY format.
+
+- a new GstMeta for closed captions: GstVideoCaptionMeta. This
+ supports the two types of closed captions, CEA-608 and CEA-708,
+ along with the four different ways they can be transported (other
+ systems are a superset of those).
+
+- a VBI (Vertical Blanking Interval) encoder for writing ancillary
+ data to the Vertical Blanking Interval lines of component signals.
+
+The new closedcaption plugin in gst-plugins-bad then makes use of all
+this new infrastructure and provides the following elements:
+
+- cccombiner: a closed caption combiner that takes a closed captions
+ stream and another stream and adds the closed captions as
+ GstVideoCaptionMeta to the buffers of the other stream.
+
+- ccextractor: a closed caption extractor which will take
+ GstVideoCaptionMeta from input buffers and output them as a separate
+ closed captions stream.
+
+- ccconverter: a closed caption converter that can convert between
+ different formats
+
+- line21decoder: extract line21 closed captions from SD video streams
+
+- cc708overlay: decodes CEA 608/708 captions and overlays them on
+ video
+
+Additionally, the following elements have also gained Closed Caption
+support:
+
+- qtdemux and qtmux support CEA 608/708 Closed Caption tracks
+
+- mpegvideoparse extracts Closed Captions from MPEG-2 video streams
+
+- decklinkvideosink can output closed captions and decklinkvideosrc
+ can extract closed captions
+
+- playbin and playbin3 learned how to autoplug CEA 608/708 CC overlay
+ elements
+
+- the externally maintained ajavideosrc element for AJA capture cards
+ has support for extracting closed captions
+
+The rsclosedcaption plugin in the Rust plugins collection includes a
+MacCaption (MCC) file parser and encoder.
+
+New Elements
+
+- overlaycomposition: New element that allows applications to draw
+ GstVideoOverlayCompositions on a stream. The element will emit the
+ "draw" signal for each video buffer, and the application then
+ generates an overlay for that frame (or not). This is much more
+ performant than e.g. cairooverlay for many use cases, e.g. because
+ pixel format conversions can be avoided or the blitting of the
+ overlay can be delegated to downstream elements (such as
+ gloverlaycompositor). It’s particularly useful for cases where only
+ a small section of the video frame should be drawn on.
+
+- gloverlaycompositor: New OpenGL-based compositor element that
+ flattens any overlays from GstVideoOverlayCompositionMetas into the
+ video stream. This element is also always part of glimagesink.
+
+- glalpha: New element that adds an alpha channel to a video stream.
+ The values of the alpha channel can either be set to a constant or
+ can be dynamically calculated via chroma keying. It is similar to
+ the existing alpha element but based on OpenGL. Calculations are
+ done in floating point so results may not be identical to the output
+ of the existing alpha element.
+
+- rtpfunnel funnels together RTP streams into a single session. Use
+ cases include multiplexing and bundle. webrtcbin uses it to
+ implement BUNDLE support.
+
+- testsrcbin is a source element that provides an audio and/or video
+ stream and also announces them using the recently-introduced
+ GstStream API. This is useful for testing elements such as playbin3
+ or uridecodebin3 etc.
+
+- New closed caption elements: cccombiner, ccextractor, ccconverter,
+ line21decoder and cc708overlay (see above)
+
+- wpesrc: new source element acting as a Web Browser based on WebKit
+ WPE
+
+- Two new OpenCV-based elements: cameracalibrate and cameraundistort
+ that can communicate to figure out distortion correction parameters
+ for a camera and correct for the distortion.
+
+- New sctp plugin based on usrsctp with sctpenc and sctpdec elements.
+ These elements are used inside webrtcbin for implementing data
+ channels.
+
+New element features and additions
+
+- playbin3, playbin and playsink have gained a new "text-offset"
+ property to adjust the positioning of the selected subtitle stream
+ vis-a-vis the audio and video streams. This uses subtitleoverlay’s
+ new "subtitle-ts-offset" property. GstPlayer has gained matching API
+ for this, namely gst_player_get_text_video_offset().
+
+- playbin3 buffering improvements: in network playback scenarios there
+ may be multiple inputs to decodebin3, and buffering will be done
+ before decodebin3 using queue2 or downloadbuffer elements inside
+ urisourcebin. Since this is before any parsers or demuxers there may
+ not be any bitrate information available for the various streams, so
+ it was difficult to configure the buffering there smartly within
+ global constraints. This was improved now: The queue2 elements
+ inside urisourcebin will now use the new bitrate query to figure out
+ a bitrate estimate for the stream if no bitrate was provided by
+ upstream, and urisourcebin will use the bitrates of the individual
+ queues to distribute the globally-set "buffer-size" budget in bytes
+ to the various queues. urisourcebin also gained "low-watermark" and
+ "high-watermark" properties which will be proxied to the internal
+ queues, as well as a read-only "statistics" property which allows
+ querying of the minimum/maximum/average byte and time levels of the
+ queues inside the urisourcebin in question.
+
+- splitmuxsink has gained a couple of new features:
+
+ - new "async-finalize" mode: This mode is useful for muxers or
+ outputs that can take a long time to finalize a file. Instead of
+ blocking the whole upstream pipeline while the muxer is doing
+ its stuff, we can unlink it and spawn a new muxer + sink
+ combination to continue running normally. This requires us to
+ receive the muxer and sink (if needed) as factories via the new
+ "muxer-factory" and "sink-factory" properties, optionally
+ accompanied by their respective properties structures (set via
+ the new "muxer-properties" and "sink-properties" properties).
+ There are also new "muxer-added" and "sink-added" signals in
+ case custom code has to be called for them to configure them.
+
+ - "split-at-running-time" action signal: When called by the user,
+ this action signal ends the current file (and starts a new one)
+ as soon as the given running time is reached. If called multiple
+ times, running times are queued up and processed in the order
+ they were given.
+
+ - "split-after" action signal to finish outputting the current GOP
+ to the current file and then start a new file as soon as the GOP
+ is finished and a new GOP is opened (unlike the existing
+ "split-now" which immediately finishes the current file and
+ writes the current GOP into the next newly-started file).
+
+ - "reset-muxer" property: when unset, the muxer is reset using
+ flush events instead of setting its state to NULL and back. This
+ means the muxer can keep state across resets, e.g. mpegtsmux
+ will keep the continuity counter continuous across segments as
+ required by hlssink2.
+
+- qtdemux gained PIFF track encryption box support in addition to the
+ already-existing PIFF sample encryption support, and also allows
+ applications to select which encryption system to use via a
+ "drm-preferred-decryption-system-id" context in case there are
+ multiple options.
+
+- qtmux: the "start-gap-threshold" property determines now whether an
+ edit list will be created to account for small gaps or offsets at
+ the beginning of a stream in case the start timestamps of tracks
+ don’t line up perfectly. Previously the threshold was hard-coded to
+ 1% of the (video) frame duration, now it is 0 by default (so edit
+ list will be created even for small differences), but fully
+ configurable.
+
+- rtpjitterbuffer has improved end-of-stream handling
+
+- rtpmp4vpay will be prefered over rtpmp4gpay for MPEG-4 video in
+ autoplugging scenarios now
+
+- rtspsrc now allows applications to send RTSP SET_PARAMETER and
+ GET_PARAMETER requests using action signals.
+
+- rtspsrc has a small (100ms) configurable teardown delay by default
+ to try and make sure an RTSP TEARDOWN request gets sent out when the
+ source element shuts down. This will block the downward PAUSED to
+ READY state change for a short time, but can be disabled where it’s
+ a problem. Some servers only allow a limited number of concurrent
+ clients, so if no proper TEARDOWN is sent new clients may have
+ problems connecting to the server for a while.
+
+- souphttpsrc behaves better with low bitrate streams now. Before it
+ would increase the read block size too quickly which could lead to
+ it not reading any data from the socket for a very long time with
+ low bitrate streams that are output live downstream. This could lead
+ to servers kicking off the client.
+
+- filesink: do internal buffering to avoid performance regression with
+ small writes since we bypass libc buffering by using writev()
+ instead of fwrite()
+
+- identity: add "eos-after" property and fix "error-after" property
+ when the element is reused
+
+- input-selector: lets context queries pass through, so that
+ e.g. upstream OpenGL elements can use contexts and displays
+ advertised by downstream elements
+
+- queue2: avoid ping-pong between 0% and 100% buffering messages if
+ upstream is pushing buffers larger than one of its limits, plus
+ performance optimisations
+
+- opusdec: new "phase-inversion" property to control phase inversion.
+ When enabled, this will slightly increase stereo quality, but
+ produces a stream that when downmixed to mono will suffer audio
+ distortions.
+
+- The x265enc HEVC encoder also exposes a "key-int-max" property to
+ configure the maximum allowed GOP size now.
+
+- decklinkvideosink has seen stability improvements for long-running
+ pipelines (potential crash due to overflow of leaked clock refcount)
+ and clock-slaving improvements when performing flushing seeks
+ (causing stalls in the output timeline), pausing and/or buffering.
+
+- srtpdec, srtpenc: add support for MKIs which allow multiple keys to
+ be used with a single SRTP stream
+
+- The srt Secure Reliable Transport plugin has integrated server and
+ client elements srt{client,server}{src,sink} into one (srtsrc and
+ srtsink), since SRT connection mode can be changed by uri
+ parameters.
+
+- h264parse and h265parse will handle SEI recovery point messages and
+ mark recovery points as keyframes as well (in addition to IDR
+ frames)
+
+- webrtcbin: "add-turn-server" action signal to pass multiple ICE
+ relays (TURN servers).
+
+- The removesilence element has received various new features and
+ properties, such as a "threshold" property, detecting silence only
+ after minimum silence time/buffers, a "silent" property to control
+ bus message notifications as well as a "squash" property.
+
+- AOMedia AV1 decoder gained support for 10/12bit decoding whilst the
+ AV1 encoder supports more image formats and subsamplings now and
+ acquired support for rate control and profile related configuration.
+
+- The Fraunhofer fdkaac plugin can now be built against the 2.0.0
+ version API and has improved multichannel support
+
+- kmssink now supports unpadded 24-bit RGB and can configure mode
+ setting from video info, which enables display of multi-planar
+ formats such as I420 or NV12 with modesetting. It has also gained a
+ number of new properties: The "restore-crtc" property does what it
+ says on the tin and is enabled by default. "plane-properties" and
+ "connector-properties" can be used to pass custom properties to the
+ DRM.
+
+- waylandsink has a "fullscreen" property now.
+
+Plugin and library moves
+
+- The stereo element was moved from -bad into the existing audiofx
+ plugin in -good. If you get duplicate type registration warnings
+ when upgrading, check that you don’t have a stale stereoplugin lying
+ about somewhere.
+
+GstVideoAggregator, compositor, and OpenGL mixer elements moved from -bad to -base
+
+GstVideoAggregator is a new base class for raw video mixers and muxers
+and is based on GstAggregator. It provides defined-latency mixing of raw
+video inputs and ensures that the pipeline won’t stall even if one of
+the input streams stops producing data.
+
+As part of the move to stabilise the API there were some last-minute API
+changes and clean-ups, but those should mostly affect internal elements.
+Most notably, the "ignore-eos" pad property was renamed to
+"repeat-after-eos" and the conversion code was moved to a
+GstVideoAggregatorConvertPad subclass to avoid code duplication, make
+things less awkward for subclasses like the OpenGL-based video mixer,
+and make the API more consistent with the audio aggregator API.
+
+It is used by the compositor element, which is a replacement for
+‘videomixer’ which did not handle live inputs very well. compositor
+should behave much better in that respect and generally behave as one
+would expected in most scenarios.
+
+The compositor element has gained support for per-pad blending mode
+operators (SOURCE, OVER, ADD) which determines what operator to use for
+blending this pad over the previous ones. This can be used to implement
+crossfading and the available operators can be extended in the future as
+needed.
+
+A number of OpenGL-based video mixer elements (glvideomixer, glmixerbin,
+glvideomixerelement, glstereomix, glmosaic) which are built on top of
+GstVideoAggregator have also been moved from -bad to -base now. These
+elements have been merged into the existing OpenGL plugin, so if you get
+duplicate type registration warnings when upgrading, check that you
+don’t have a stale openglmixers plugin lying about somewhere.
+
+Plugin removals
+
+The following plugins have been removed from gst-plugins-bad:
+
+- The experimental daala plugin has been removed, since it’s not so
+ useful now that all effort is focused on AV1 instead, and it had to
+ be enabled explicitly with --enable-experimental anyway.
+
+- The spc plugin has been removed. It has been replaced by the gme
+ plugin.
+
+- The acmmp3dec and acmenc plugins for Windows have been removed. ACM
+ is an ancient legacy API and there was no point in keeping the
+ plugins around for a licensed MP3 decoder now that the MP3 patents
+ have expired and we have a decoder in -good. We also didn’t ship
+ these in our cerbero-built Windows packages, so it’s unlikely that
+ they’ll be missed.
+
+
+Miscellaneous API additions
+
+- GstBitwriter: new generic bit writer API to complement the existing
+ bit reader
+
+- gst_buffer_new_wrapped_bytes() creates a wrap buffer from a GBytes
+
+- gst_caps_set_features_simple() sets a caps feature on all the
+ structures of a GstCaps
+
+- New GST_QUERY_BITRATE query: This allows determining from downstream
+ what the expected bitrate of a stream may be which is useful in
+ queue2 for setting time based limits when upstream does not provide
+ timing information. tsdemux, qtdemux and matroskademux have basic
+ support for this query on their sink pads.
+
+- elements: there is a new “Hardware” class specifier. Elements
+ interacting with hardware devices should specify this classifier in
+ their element factory class metadata. This is useful to advertise as
+ one might need to put such elements into READY state to test if the
+ hardware is present in the system for example.
+
+- protection: Add a new definition for unspecified system protection,
+ GST_PROTECTION_UNSPECIFIED_SYSTEM_ID
+
+- take functions for various mini objects that didn’t have them yet:
+ gst_query_take(), gst_message_take(), gst_tag_list_take(),
+ gst_buffer_list_take(). Unlike the various _replace() functions
+ _take() does not increase the reference count but takes ownership of
+ the mini object passed.
+
+- clear functions for various mini object types and GstObject which
+ unrefs the object or mini object (if non-NULL) and sets the variable
+ pointed to to NULL: gst_clear_structure(), gst_clear_tag_list(),
+ gst_clear_query(), gst_clear_message(), gst_clear_event(),
+ gst_clear_caps(), gst_clear_buffer_list(), gst_clear_buffer(),
+ gst_clear_mini_object(), gst_clear_object()
+
+- miniobject: new API gst_mini_object_add_parent() and
+ gst_mini_object_remove_parent() to set parent pointers on mini
+ objects to ensure correct writability: Every container of
+ miniobjects now needs to store itself as parent in the child object,
+ and remove itself again later. A mini object is then only writable
+ if there is at most one parent, that parent is writable itself, and
+ the reference count of the mini object is 1. GstBuffer (for
+ memories), GstBufferList (for buffers), GstSample (for caps, buffer,
+ bufferlist), and GstVideoOverlayComposition were updated
+ accordingly. Without this it was possible to have e.g. a buffer list
+ with a refcount of 2 used in two places at once that both modify the
+ same buffer with refcount 1 at the same time wrongly thinking it is
+ writable even though it’s really not.
+
+- poll: add API to watch for POLLPRI and stop treating POLLPRI as a
+ read. This is useful to wait for video4linux events which are
+ signalled via POLLPRI.
+
+- sample: new API to update the contents of a GstSample and make it
+ writable: gst_sample_set_buffer(), gst_sample_set_caps(),
+ gst_sample_set_segment(), gst_sample_set_info(), plus
+ gst_sample_is_writable() and gst_sample_make_writable(). This makes
+ it possible to reuse a sample object and avoid unnecessary memory
+ allocations, for example in appsink.
+
+- ClockIDs now keep a weak reference to underlying clock to avoid
+ crashes in basesink in corner cases where a clock goes away while
+ the ClockID is still in use, plus some new API
+ (gst_clock_id_get_clock(), gst_clock_id_uses_clock()) to check the
+ clock a ClockID is linked to.
+
+- The GstCheck unit test library gained a
+ fail_unless_equals_clocktime() convenience macro as well as some new
+ GstHarness API for for proposing meta APIs from the allocation
+ query: gst_harness_add_propose_allocation_meta(). ASSERT_CRITICAL()
+ checks in unit tests are now skipped if GStreamer was compiled with
+ GST_DISABLE_GLIB_CHECKS.
+
+- gst_audio_buffer_truncate() convenience function to truncate a raw
+ audio buffer
+
+
+Miscellaneous performance and memory optimisations
+
+As always there have been many performance and memory usage improvements
+across all components and modules. Some of them (such as dmabuf
+import/export) have already been mentioned elsewhere so won’t be
+repeated here.
+
+The following list is only a small snapshot of some of the more
+interesting optimisations that haven’t been mentioned in other contexts
+yet:
+
+- The GstVideoEncoder and GstVideoDecoder base classes now release the
+ STREAM_LOCK when pushing out buffers, which means (multi-threaded)
+ encoders and decoders can now receive and continue to process input
+ buffers whilst waiting for downstream elements in the pipeline to
+ process the buffer that was pushed out. This increases throughput
+ and reduces processing latency, also and especially for
+ hardware-accelerated encoder/decoder elements.
+
+- GstQueueArray has seen a few API additions
+ (gst_queue_array_peek_nth(), gst_queue_array_set_clear_func(),
+ gst_queue_array_clear()) so that it can be used in other places like
+ GstAdapter instead of a GList, which reduces allocations and
+ improves performance.
+
+- appsink now reuses the sample object in pull_sample() if possible
+
+- rtpsession only starts the RTCP thread when it’s actually needed now
+
+- udpsrc uses a buffer pool now and the GstUdpSrc object structure was
+ optimised for better cache performance
+
+GstPlayer
+
+- API was added to fine-tune the synchronisation offset between
+ subtitles and video
+
+
+Miscellaneous changes
+
+- As a result of moving to newer FFmpeg APIs, encoder and decoder
+ elements exposed by the GStreamer FFmpeg wrapper plugin (gst-libav)
+ may have seen possibly incompatible changes to property names and/or
+ types, and not all properties exposed might be functional. We are
+ still reviewing the new properties and aim to minimise breaking
+ changes at least for the most commonly-used properties, so please
+ report any issues you run into!
+
+OpenGL integration
+
+- The OpenGL mixer elements have been moved from -bad to
+ gst-plugins-base (see above)
+
+- The Mesa GBM backend now supports headless mode
+
+- gloverlaycompositor: New OpenGL-based compositor element that
+ flattens any overlays from GstVideoOverlayCompositionMetas into the
+ video stream.
+
+- glalpha: New element that adds an alpha channel to a video stream.
+ The values of the alpha channel can either be set to a constant or
+ can be dynamically calculated via chroma keying. It is similar to
+ the existing alpha element but based on OpenGL. Calculations are
+ done in floating point so results may not be identical to the output
+ of the existing alpha element.
+
+- glupload: Implement direct dmabuf uploader, the idea being that some
+ GPUs (like the Vivante series) can actually perform the YUV->RGB
+ conversion internally, so no custom conversion shaders are needed.
+ To make use of this feature, we need an additional uploader that can
+ import DMABUF FDs and also directly pass the pixel format, relying
+ on the GPU to do the conversion.
+
+
+Tracing framework and debugging improvements
+
+- There is now a GDB PRETTY PRINTER FOR VARIOUS GSTREAMER TYPES: For
+ GstObject pointers the type and name is added, e.g.
+ 0x5555557e4110 [GstDecodeBin|decodebin0]. For GstMiniObject pointers
+ the object type is added, e.g. 0x7fffe001fc50 [GstBuffer]. For
+ GstClockTime and GstClockTimeDiff the time is also printed in human
+ readable form, e.g. 150116219955 [+0:02:30.116219955].
+
+- GDB EXTENSION WITH TWO CUSTOM GDB COMMANDS gst-dot AND gst-print:
+
+ - gst-dot creates dot files that a very close to what
+ GST_DEBUG_BIN_TO_DOT_FILE() produces, but object properties and
+ buffer contents such as codec-data in caps are not available.
+
+ - gst-print produces high-level information about a GStreamer
+ object. This is currently limited to pads for GstElements and
+ events for the pads. The output may look like this:
+
+ (gdb) gst-print pad.object.parent
+ GstMatroskaDemux (matroskademux0) {
+ SinkPad (sink, pull) {
+ }
+ SrcPad (video_0, push) {
+ events:
+ stream-start:
+ stream-id: 0463ccb080d00b8689bf569a435c4ff84f9ff753545318ae2328ea0763fd0bec/001:1274058367
+ caps: video/x-theora
+ width: 1920
+ height: 800
+ pixel-aspect-ratio: 1/1
+ framerate: 24/1
+ streamheader: < 0x5555557c7d30 [GstBuffer], 0x5555557c7e40 [GstBuffer], 0x7fffe00141d0 [GstBuffer] >
+ segment: time
+ rate: 1
+ tag: global
+ container-format: Matroska
+ }
+ SrcPad (audio_0, push) {
+ events:
+ stream-start:
+ stream-id: 0463ccb080d00b8689bf569a435c4ff84f9ff753545318ae2328ea0763fd0bec/002:1551204875
+ caps: audio/mpeg
+ mpegversion: 4
+ framed: true
+ stream-format: raw
+ codec_data: 0x7fffe0014500 [GstBuffer]
+ level: 2
+ base-profile: lc
+ profile: lc
+ channels: 2
+ rate: 44100
+ segment: time
+ rate: 1
+ tag: global
+ container-format: Matroska
+ tag: stream
+ audio-codec: MPEG-4 AAC audio
+ language-code: en
+ }
+ }