ffv1: Make sure at least one slice context is initialized

This avoids crashes when initializing the range coder for
the first slice context.

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
Signed-off-by: Martin Storsjö <martin@martin.st>
(cherry picked from commit b1db33159fdc2da4bdd8c75e4ff9a7dd0ef2f0c2)
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

truemotion2: Use av_freep properly in an error path

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
Signed-off-by: Martin Storsjö <martin@martin.st>
(cherry picked from commit c39f7eba01cd656e8f0eed592f93d11814736650)
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
(cherry picked from commit eac1c3f384eab770d42468f4f244156c1735701d)

eacmv: Make sure a reference frame exists before referencing it

This is similar to an existing check for the second-last frame
from 062421e3.

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
Signed-off-by: Martin Storsjö <martin@martin.st>
(cherry picked from commit ea78a348d86a3a733f6c1e0a65cfdd8283d924b9)
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
Conflicts:
libavcodec/eacmv.c
(cherry picked from commit 2e12af4587613dd5b2c3431e5c8194d73b03434f)

mpeg4videodec: Check the width/height in mpeg4_decode_sprite_trajectory

This avoids a potential division by zero.

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
Signed-off-by: Martin Storsjö <martin@martin.st>
(cherry picked from commit f875a732e36786d49f3650e3235272891a820600)
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
(cherry picked from commit c8c93795e4afd04c2c5b74e29e8dec29b6a76b81)

ivi_common: Make sure color planes have been initialized

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
Signed-off-by: Martin Storsjö <martin@martin.st>
(cherry picked from commit a92538b7c0defc86c55fb91f55dfa36aad192673)
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
(cherry picked from commit 38bd229af9c4fa5897fc1a69e73a04c55f78647f)

oggparseogm: Convert to use bytestream2

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
Signed-off-by: Martin Storsjö <martin@martin.st>
(cherry picked from commit 19b9659f3174599e8685d329c4330b1ea8c4c6db)
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

rv34: Check the return value from ff_rv34_decode_init

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
Signed-off-by: Martin Storsjö <martin@martin.st>
(cherry picked from commit 711c970168297683860422e95d6b7e37ee3c8367)
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
(cherry picked from commit 20c8f176293e7520c6205b664e25ecf8a711253e)

matroskadec: Verify realaudio codec parameters

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
Signed-off-by: Martin Storsjö <martin@martin.st>
(cherry picked from commit 569d18aa9dc989c37bb4d4b968026fe5afa6fff9)
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
(cherry picked from commit 9f7a8b8f8f6ad024410232d926b774261ef2ef36)

mace: Make sure that the channel count is set to a valid value

Also return a proper error code.

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
Signed-off-by: Martin Storsjö <martin@martin.st>
(cherry picked from commit e1f3847f860a1094a46be4c5f10db8df616c3135)
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
Conflicts:
libavcodec/mace.c

svq3: Check for any negative return value from ff_h264_check_intra_pred_mode

Also pass on any returned error code.

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
Signed-off-by: Martin Storsjö <martin@martin.st>
(cherry picked from commit 1115689d54ea95a084421f5a182b8dc56cbff978)
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
Conflicts:
libavcodec/svq3.c

vp3: Check the framerate for validity

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
Signed-off-by: Martin Storsjö <martin@martin.st>
(cherry picked from commit 6fc8226e29055858f28973bb3d27b63b3b65e616)
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
(cherry picked from commit b4c479a82adbb1301e3e549cd80cdd65208ddd05)

cavsdec: Make sure a sequence header has been decoded before decoding pictures

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
Signed-off-by: Martin Storsjö <martin@martin.st>
(cherry picked from commit e90a6846c2c006fbebd00e1f2789f4a86fafacef)
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
Conflicts:
libavcodec/cavsdec.c

sierravmd: Do sanity checking of frame sizes

Limit the size to INT_MAX/2 (for simplicity) to be sure that
size + BYTES_PER_FRAME_RECORD won't overflow.

Also factorize other existing error return paths.

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
Signed-off-by: Martin Storsjö <martin@martin.st>
(cherry picked from commit 0ef1660a6365ce60ead8858936b6f3f8ea862826)
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
(cherry picked from commit 153deed18bed43d16b272e8681b2a9b988d2682a)

omadec: Properly check lengths before incrementing the position

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
Signed-off-by: Martin Storsjö <martin@martin.st>
(cherry picked from commit 342c43d154e586bc022c86b168fe8d36f69da9d3)
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
(cherry picked from commit 9eba02d5dd7036294ea350cb772822deec95b867)

mpc8: Make sure the first stream exists before parsing the seek table

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
Signed-off-by: Martin Storsjö <martin@martin.st>
(cherry picked from commit 17d57848fc14e82f76a65ffb25c90f2f011dc4a0)
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
(cherry picked from commit 557df77eab7d3726c34221aeb999afe9e7818d52)

mpc8: Check the seek table size parsed from the bitstream

Limit the size to INT_MAX/2 (for simplicity) to be sure that
size + FF_INPUT_BUFFER_PADDING_SIZE won't overflow.

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
Signed-off-by: Martin Storsjö <martin@martin.st>
(cherry picked from commit 459f2b393a3f89ed08d10fbceb4738d1429f268e)
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
(cherry picked from commit f8a72f041c049e812dfa1f32156327e9778f5710)

zmbvdec: Check the buffer size for uncompressed data

Also don't pointlessly set the buffer size to 1 after copying
one packet.

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
Signed-off-by: Martin Storsjö <martin@martin.st>
(cherry picked from commit 0d61f260010707f3028b818e8b24598e1a83d696)
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

ape: Don't allow the seektable to be omitted

The seektable is required for filling in ape->frames[i].pos
further down.

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
Signed-off-by: Martin Storsjö <martin@martin.st>
(cherry picked from commit 183b9d843a9533774fabd3984a52f3987001acbc)
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

shorten: Break out of loop looking for fmt chunk if none is found

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
Signed-off-by: Martin Storsjö <martin@martin.st>
(cherry picked from commit b26742cc308552f242ee2bf93b07a3ff509f4edc)
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

shorten: Use a checked bytestream reader for the wave header

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
Signed-off-by: Martin Storsjö <martin@martin.st>
(cherry picked from commit 49568851bf1700e3d9ea9cda29208d0df3c2c38b)
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

smacker: Make sure we don't fill in huffman codes out of range

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
Signed-off-by: Martin Storsjö <martin@martin.st>
(cherry picked from commit 0679cec6e8802643bbe6d5f68ca1110a7d3171da)
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

smacker: Avoid integer overflow when allocating packets

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
Signed-off-by: Martin Storsjö <martin@martin.st>
(cherry picked from commit 710b0e27025948b7511821c2f888ff2d74a59e14)
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

smacker: Don't return packets in unallocated streams

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
Signed-off-by: Martin Storsjö <martin@martin.st>
(cherry picked from commit 8d928023f953a28692ba27071a448259134b103b)
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

dsicin: Add some basic sanity checks for fields read from the file

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
Signed-off-by: Martin Storsjö <martin@martin.st>
(cherry picked from commit 48d6556dd46d4f4fac10d0f4a819e314887cd50e)
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

arm: Don't clobber callee saved registers in scalarproduct

q4-q7/d8-d15 are supposed to not be clobbered by the callee.

CC: libav-stable@libav.org
Signed-off-by: Martin Storsjö <martin@martin.st>
(cherry picked from commit d307e408d4a9ada22df443cc38be77cc5e492694)

Signed-off-by: Martin Storsjö <martin@martin.st>

Prepare for 0.8.10 Release

roqvideodec: check dimensions validity

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 3ae610451170cd5a28b33950006ff0bd23036845)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit fee26d352a52eb9f7fcd8d9167fb4a5ba015b612)

CC: libav-stable@libav.org
Signed-off-by: Reinhard Tartler <siretart@tauware.de>
(cherry picked from commit 488f87be873506abb01d67708a67c10a4dd29283)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
(cherry picked from commit 52b18c1fde65efac7f6e6104b76d39bf8d0a34ee)
Signed-off-by: Reinhard Tartler <siretart@tauware.de>

qdm2: check array index before use, fix out of array accesses

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit a7ee6281f7ef1c29284e3a4cadfe0f227ffde1ed)

CC: libav-stable@libav.org
Signed-off-by: Reinhard Tartler <siretart@tauware.de>
(cherry picked from commit 39bec05ed42e505d17877b0c23f16322f9b5883b)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
(cherry picked from commit 0b2b8ab979624b0cce673d5e99255482d7c553ad)
Signed-off-by: Reinhard Tartler <siretart@tauware.de>

alsdec: check block length

Fix writing over the end

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Addresses: CVE-2013-0845
(cherry picked from commit 2a0fb7286d67c47e44aa76c237ede117b22af616)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
(cherry picked from commit 3f7d89034bfe50893927cc92ddcb95a2e9b4178d)
Signed-off-by: Reinhard Tartler <siretart@tauware.de>

Changelog for 0.8.9

Prepare for 0.8.7 Release

x86: fft: Remove 3DNow! optimizations, they break FATE

x86: ac3dsp: Drop mmx variant of ac3_max_msb_abs_int16

The function accidentally uses mmxext instructions, so it causes sigill
on mmx-only CPUs and provides no benefit on CPUs with mmxext available.

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

aac: Check init_get_bits return value

Some code paths can call it with invalid length.

CC: libav-stable@libav.org
(cherry picked from commit 71953ebcf94fe4ef316cdad1f276089205dd1d65)
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

aac: return meaningful errors

(cherry picked from commit 07c52e2c7c60b087fd023cd9771778973def0b33)
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
Conflicts:
libavcodec/aacdec.c

dsicinav: K&R formatting cosmetics

(cherry picked from commit fcae3ff124ee97c9265e3b93f3d41238b2aee9bd)
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
Conflicts:
libavcodec/dsicinav.c

mov: Seek back if overreading an individual atom

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Cc: libav-stable@libav.org
Signed-off-by: Martin Storsjö <martin@martin.st>
(cherry picked from commit 5b4eb243bce10a3e8345401a353749e0414c54ca)

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
Conflicts:
libavformat/mov.c

vcr1: add sanity checks

Fixes invalid reads with corrupted files.

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 8aba7968dd604aae91ee42cbce0be3dad7dceb30)
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
Conflicts:
libavcodec/vcr1.c

pictordec: pass correct context to avpriv_request_sample

Fixes invalid reads.

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry-picked from commit fe9bb61f9a16be19ad91875632c39e44b7a99a8a)
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
Conflicts:
libavcodec/pictordec.c

dsicinav: Clip the source size to the expected maximum

A packet larger than cin->bitmap_size does not make sense.

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit fd8189932147a524fe43532b46baa35e8be92a1b)

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
Conflicts:
libavcodec/dsicinav.c

alsdec: Clean up error paths

Fix at least a memory leak.

CC: libav-stable@libav.org
(cherry picked from commit ca488ad480360dfafcb5766f7bfbb567a0638979)

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
Conflicts:
libavcodec/alsdec.c

ogg: Fix potential infinite discard loop

Seeking in certain broken files would cause ogg_read_timestamp
to fail because ogg_packet would go into a state where all packets
of stream 1 would be discarded until the end of the stream.

Bug-Id: 553
CC: libav-stable@libav.org
Signed-off-by: Jan Gerber <j@v2v.cc>
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
(cherry picked from commit 9a27acae9e6b7d0bf74c5b878af9c42495a546f3)
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
Conflicts:
libavformat/oggdec.c

nuv: check rtjpeg_decode_frame_yuv420 return value

CC: libav-stable@libav.org
(cherry picked from commit 85ac12587bfef970d0e0e4abc292df346daf8478)
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
Conflicts:
libavcodec/nuv.c

nuv: Reset the frame on resize

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
Conflicts:
libavcodec/nuv.c

nuv: Use av_fast_realloc

The decompressed buffer can be used after codec_reinit, so it must be
preserved.

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 2df0776c2293efb0ac12c003843ce19332342e01)
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
Conflicts:
libavcodec/nuv.c

nuv: return meaningful error codes.

(cherry picked from commit 3344f5cb747bb1f54cc34878b66dc0536f194720)
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
Conflicts:
libavcodec/nuv.c

nuv: Pad the lzo outbuf

And properly update the buf_size with the correct size.

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 075dbc185521f193c98b896cd63be3ec2613df5d)
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
Conflicts:
libavcodec/nuv.c

nuv: Do not ignore lzo decompression failures

Update the fate reference since the last broken frame is not decoded
anymore.

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit aae159a7cc4df7d0521901022b778c9da251c24e)
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
Conflicts:
libavcodec/nuv.c

oma: correctly mark and decrypt partial packets

Incomplete crypted files would lead to a read after buffer boundary
otherwise.

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 2219e27b5b17d146e4ab71a3ed86dfc013fb7a93)
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
Conflicts:
libavformat/omadec.c

oma: check geob tag boundary

Prevent read after buffer boundary on corrupted tag.

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 9d0b45ade864f3d2ccd8610149fe1fff53c4e937)
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
Conflicts:
libavformat/omadec.c

oma: refactor seek function

Properly propagate seek errors from avio and the generic pcm seek.

(cherry picked from commit 4f03a77e52596cbe9ec179666ddb3e0345a8133a)
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
Conflicts:
libavformat/omadec.c

8bps: Bound-check the input buffer

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit bd7b4da0f4627bb6c4a7c2575da83fe6b261a21c)

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
Conflicts:
libavcodec/8bps.c

rtmp: Do not misuse memcmp

CC: libav-stable@libav.org
(cherry picked from commit 5718e3487ba3b26aba341070be0b6b0b4de45ea3)
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
Conflicts:
libavformat/rtmppkt.h
libavformat/rtmpproto.c

rtmp: rename data_size to size

(cherry picked from commit ba5393a609c723ec8ab7f9727c10fef734c09278)
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
Conflicts:
libavformat/rtmppkt.c
libavformat/rtmpproto.c

lavc: set the default rc_initial_buffer_occupancy

rc_buffer_size is not set before.

Solve the initial the rate control underflow issue reported in
bug 222.

CC: libav-stable@libav.org
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
(cherry picked from commit bff3607547fdbb6e32b3830a351e6a33280c1e0d)

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

4xm: Reject not a multiple of 16 dimension

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 2f034f255c49050e894ab9b88087c09ebe249f3f)

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

4xm: do not overread the prestream buffer

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit be373cb50d3c411366fec7eef2eb3681abe48f96)

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

4xm: validate the buffer size before parsing it

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit de2e5777e225e75813daf2373c95e223651fd89a)

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

indeo: Do not reference mismatched tiles

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit f9e5261cab067be7278f73d515bc9b601eb56202)

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

indeo: Sanitize ff_ivi_init_planes fail paths

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 28dda8a691f1c723a4a9365ab85f9625f1330096)

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

indeo: Bound-check before applying motion compensation

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 25a6666f6c07c6ac8449a63d7fbce0dfd29c54cd)

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

indeo: Bound-check before applying transform

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit dc79685195a45c9b8b17d7b93d118e0aefa45462)

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
Conflicts:
libavcodec/ivi_common.c

indeo: reject negative array indexes

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 6a10142faa1cca8ba2bfe51b970754f62d60f320)

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

indeo: Cosmetic formatting

Trim some overly long lines.

(cherry picked from commit 6dfacd7ab126aea1392949d1aa10fdc3d3eeb911)

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
Conflicts:
libavcodec/ivi_common.c

indeo: Refactor ff_ivi_init_tiles and ivi_decode_blocks

Spin large and mostly self contained blocks into stand alone
functions.

(cherry picked from commit 62256010e9bc8879e2bf7f3b94af8ff85e239082)

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

indeo: Refactor ff_ivi_dec_huff_desc

Spare an indentation level.

(cherry picked from commit f6f36ca8ca1b2526d3abff7d7c627322d3bce912)

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

lavf: fix the comparison in an overflow check

CC: libav-stable@libav.org
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
(cherry picked from commit 26f027fba1c5ab482fa2488fbe0fa36c8bb33b69)
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

dv: Add a guard to not overread the ppcm array

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 7ee191cab0dc44700f26c5784e2adeb6a779651b)
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
Conflicts:
libavformat/dv.c

mpegvideo: Avoid 32-bit wrapping of linesize multiplications

This makes sure that linesize * start_y doesn't overflow, so that
emulated_edge_mc can get back the original value if needed.

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
Signed-off-by: Martin Storsjö <martin@martin.st>
(cherry picked from commit a711a2cb473dc95708f371a82c85c97fe789b5c2)
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

mjpegb: Detect changing number of planes in interlaced video

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
Signed-off-by: Martin Storsjö <martin@martin.st>
(cherry picked from commit af11fa5409cc72fc45ca7f3527400beca10967b9)
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

matroskadec: Check that .lang was allocated and set before reading it

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
Signed-off-by: Martin Storsjö <martin@martin.st>
(cherry picked from commit 5bcd3ae5b167fb74215520b01d5d810e0c8986ab)
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

ape demuxer: check for EOF in potentially long loops

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry-picked from commit 488b2984fece7ad0c2596826fee18e74aa904667)
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

lavf: avoid integer overflow when estimating bitrate

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit df33a58e5311ee9a64a573889b883a80e981af7b)
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

pictordec: break out of both decoding loops when y drops below 0

Otherwise picmemset can get called with negative y, resulting in an
invalid write.

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 5f7aecde02a95451e514c809f2794c1deba80695)
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

ac3: Return proper error codes

(cherry picked from commit b1f9cdc37ff5d5b391d2cd9af737ab4e5a0fc1c0)
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

ac3: Clean up the error paths

(cherry picked from commit 818d1f1a3e89d35213af0bd5dc4a772713951882)
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

ac3: Do not clash with normal AVERROR

The parsing function return AVERROR and AAC_AC3_PARSE_ERROR values,
make sure they are not misunderstood.

(cherry picked from commit 6258d362b82934a2c27557e0984aed372d98091a)
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

dxa: Make sure the reference frame exists

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 5ef7c84a9374681c64722a96d91741f3b990af2b)
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
Conflicts:
libavcodec/dxa.c

h261: check the mtype index

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit c59967fa7cc5bc2fa06b36c17d2c207240c06b3e)
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
Conflicts:
libavcodec/h261dec.c

segafilm: Error out on impossible packet size

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 5268bd2900effa59b51e0fede61aacde5e2f0b95)
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

ogg: Always alloc the private context in vorbis_header

It is possible to have an initial broken header and then valid packets.

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 3562684db716d11de0b0dcc52748e9cd90d68132)
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

vc1: check mb_height validity.

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 43bacd5b7d3d265a77cd29d8abb131057796aecc)
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

vc1: check the source buffer in vc1_mc functions

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 090cd0631140ac1a3a795d2adfac5dbf5e381aa2)
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
Conflicts:
libavcodec/vc1dec.c

bink: Bound check the quantization matrix.

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 9991298f2c4d9022ad56057f15d037e18d454157)
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

xl: Make sure the width is valid

CC: libav-stable@libav.org
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

alsdec: Fix the clipping range

mcc_weightings is only 32 elements.

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 70ecc175c7b513a153ac87d1c5d219556ca55070)

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

dsicinav: Bound-check the source buffer when needed

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit dd0bfc3a6a310e3e3674ce7742672d689a9a0e93)

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

mov: Do not allow updating the time scale after it has been set

The time scale is set in mdhd, and later validated in the
enclosing trak atom once all of its children have been parsed.

A loose mdhd atom outside of a trak atom could update the time
scale of the last stream without any validation.

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Cc: libav-stable@libav.org
Signed-off-by: Martin Storsjö <martin@martin.st>
(cherry picked from commit 31931520df35a6f9606fe8293c8a39e2d1fabedf)

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

ac3dec: Don't consume more data than the actual input packet size

This was handled properly in the normal return case at the end
of the function, but not in this special case.

Returning a value larger than the input packet size can cause
problems for certain library users.

Returning the actual input buffer size unconditionally, since
it is not guaranteed that frame_size is set to a sensible
value at this point.

Cc: libav-stable@libav.org
Signed-off-by: Martin Storsjö <martin@martin.st>
(cherry picked from commit 8f24c12be7a3b3ea105e67bba9a867fe210a2333)

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

indeo: Reject impossible FRAMETYPE_NULL

A frame marked FRAMETYPE_NULL cannot be scalable and requires a
previous frame successfully decoded.

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 5b2a29552ca09edd4646b6aa1828b32912b7ab36)

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

indeo5: return proper error codes

(cherry picked from commit b0eeb9d442e4b7e82f6797d74245434ea33110a5)

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

indeo4: Validate scantable dimension

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit cd78e934c246d1b2510f8fba0abfe40bb75795f6)

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

indeo4: Check the quantization matrix index

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 6255ccf7d51c82ab79bf0cd47a921f572dda4489)

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

indeo4: Do not access missing reference MV

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 8435bca087c0e79385763c51de009fd89390b6a5)

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
Conflicts:

libavcodec/indeo4.c

adpcm: Unbreak ima-dk4

Was broken by commit b9dea1a085c4705e480bd17dfa8c8ce227fdce76

ac3dec: validate channel output mode against channel count

Damaged frames can lead to a mismatch, which can cause a segfault
due to using an incorrect channel mapping.

CC:libav-stable@libav.org
(cherry picked from commit d7c450436fcb9d3ecf59884a574e7684183e753d)

Conflicts:

libavcodec/ac3dec.c

dca: Respect the current limits in the downmixing capabilities

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 3802833bc1f79775a1547c5e427fed6e92b77e53)

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

dca: Error out on missing DSYNC

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit f261e508459e28beca59868a878e1519a44bb678)

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

pcm: always use codec->id instead of codec_id

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit c82da343e635663605bd81c59d872bee3182da73)

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
Conflicts:
libavcodec/pcm.c

mlpdec: Do not set invalid context in read_restart_header

The faulty values rippled further down the codepath causing a
hard-to-track segfault in the assembly code.

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit e9d394f3fad7e8fd8fc80e3b33cb045bbaceb446)

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
Conflicts:
libavcodec/mlpdec.c