swscale: fix overflows in output of RGB48 pixels.
authorRonald S. Bultje <rsbultje@gmail.com>
Fri, 16 Dec 2011 00:03:54 +0000 (00:03 +0000)
committerMans Rullgard <mans@mansr.com>
Sat, 17 Dec 2011 18:36:20 +0000 (18:36 +0000)
commitbe1bafc303a77fa9ca5a99e36a65d5765012d3f4
tree6e7daa2b038cb79f72f8d0fc4ef04756eebad93d
parent5d8122db5c0b537c4d2c3352b4c89cb92f865bc2
swscale: fix overflows in output of RGB48 pixels.

For certain types of filters where the intermediate sum of coefficients
can go above the fixed-point equivalent of 1.0 in the middle of a filter,
the sum of a 31-bit calculation can overflow in both directions and can
thus not be represented in a 32-bit signed or unsigned integer. To work
around this, we subtract 0x40000000 from a signed integer base, so that
we're halfway signed/unsigned, which makes it fit even if it overflows.
After the filter finishes, we add the scaled bias back after a shift.

We use the same trick for 16-bit bpc YUV output routines.

Signed-off-by: Mans Rullgard <mans@mansr.com>
libswscale/swscale.c