rtpenc: fix overflow checking in avc_mp4_find_startcode()
authorXi Wang <xi.wang@gmail.com>
Wed, 23 Jan 2013 01:58:07 +0000 (20:58 -0500)
committerMartin Storsjö <martin@martin.st>
Wed, 23 Jan 2013 11:51:29 +0000 (13:51 +0200)
commitcf29f49d8ae00bb153c24b5c8a8f6cb150a91de8
tree8aa856c0dee1acd12df604bfb49792405bec66f9
parentecb918e5f0a4395468862b5fbd11a51de9be3d4f
rtpenc: fix overflow checking in avc_mp4_find_startcode()

The check `start + res < start' is broken since pointer overflow is
undefined behavior in C.  Many compilers such as gcc/clang optimize
away this check.

Use `res > end - start' instead.  Also change `res' to unsigned int
to avoid signed left-shift overflow.

Signed-off-by: Xi Wang <xi.wang@gmail.com>
Signed-off-by: Martin Storsjö <martin@martin.st>
libavformat/rtpenc_h264.c