fix signed integer overflow of idct
authorLinfeng Zhang <linfengz@google.com>
Tue, 26 Sep 2017 19:33:40 +0000 (12:33 -0700)
committerJames Zern <jzern@google.com>
Wed, 27 Sep 2017 18:17:54 +0000 (11:17 -0700)
commitdbbbd4430411ba8931de18a8e802934de511c22c
treeafad0168ee402432a9eb6407ca4daf93b97b4d63
parent9e52d3910bcb36789bc04b9be2db10c7c3fc459f
fix signed integer overflow of idct

Exposed by fuzz test in high bitdepth.
The bug is introduced in commit 64653fa.

BUG=webm:1466

Change-Id: Idd77d5c6a60efb9241471611ce1aba0646cb6ff5
test/invalid_file_test.cc
test/test-data.mk
test/test-data.sha1
vpx_dsp/arm/idct16x16_1_add_neon.c
vpx_dsp/arm/idct32x32_1_add_neon.c
vpx_dsp/arm/idct4x4_1_add_neon.c
vpx_dsp/arm/idct8x8_1_add_neon.c
vpx_dsp/inv_txfm.c
vpx_dsp/x86/inv_txfm_sse2.c