Change highbd variance rounding to prevent negative variance.
authorAlex Converse <aconverse@google.com>
Fri, 20 Nov 2015 21:48:51 +0000 (13:48 -0800)
committerAlex Converse <aconverse@google.com>
Wed, 25 Nov 2015 00:32:01 +0000 (16:32 -0800)
commit022c848b4d04a2c1f56a54748eee503767269023
tree3a27a7582fb2719361e679c5ad738c69db8193cb
parent64a60ce3ba78c33a69e0adf23f68638d644015e3
Change highbd variance rounding to prevent negative variance.

Always round sum error and sum square error toward zero in variance
calculations. This prevents variance from becoming negative.
Avoiding rounding variance at all might be better but would be far
more invasive.

Change-Id: Icf24e0e75ff94952fc026ba6a4d26adf8d373f1c
test/variance_test.cc
vpx_dsp/variance.c
vpx_dsp/variance.h
vpx_dsp/x86/highbd_variance_sse2.c