Fix using representable texture channel ranges.

A value of 4294967295 (2^32 - 1) is not exactly representable in
IEEE-754 single-precision floating-point format, and so it gets rounded
to the next representable value, which is 4294967296. However, this
value can't be cast to an unsigned 32-bit integer without overflowing.

GLSL does not define what happens on overflow, and IEEE-754 defines it
as an exception but GLSL forbids exceptions. Hence some implementations
may produce unexpected results. dEQP assumed clamping to the largest
representable integer.

This change fixes that false assumption by reducing the range to values
representable in both float and integer formats.

Note that 32-bit integer formats can still hold values slightly larger
than these ranges. So while previously the floating-point ranges were
too large to represent integer values, they are now too small. This
can't be fixed without separating the integer format tests and only
using integer values to represent their ranges. This doesn't appear
necessary for the time being since the tests that use these floating-
point ranges have large 12/256 tolerances for the output color.

Google bug: 70910885
Component: Framework

diff --git a/framework/common/tcuTextureUtil.cpp b/framework/common/tcuTextureUtil.cpp
index 48e49e2..215442b 100644 (file)
--- a/framework/common/tcuTextureUtil.cpp
+++ b/framework/common/tcuTextureUtil.cpp
@@ -381,11 +381,11 @@ static Vec2 getFloatChannelValueRange (TextureFormat::ChannelType channelType)
                // Misc formats.
                case TextureFormat::SIGNED_INT8:                                        cMin = -128.0f;                 cMax = 127.0f;                  break;
                case TextureFormat::SIGNED_INT16:                                       cMin = -32768.0f;               cMax = 32767.0f;                break;
-               case TextureFormat::SIGNED_INT32:                                       cMin = -2147483648.0f;  cMax = 2147483647.0f;   break;
+               case TextureFormat::SIGNED_INT32:                                       cMin = -2147483520.0f;  cMax = 2147483520.0f;   break; // Maximum exactly representable 31-bit integer: (2^24 - 1) * 2^7
                case TextureFormat::UNSIGNED_INT8:                                      cMin = 0.0f;                    cMax = 255.0f;                  break;
                case TextureFormat::UNSIGNED_INT16:                                     cMin = 0.0f;                    cMax = 65535.0f;                break;
                case TextureFormat::UNSIGNED_INT24:                                     cMin = 0.0f;                    cMax = 16777215.0f;             break;
-               case TextureFormat::UNSIGNED_INT32:                                     cMin = 0.0f;                    cMax = 4294967295.f;    break;
+               case TextureFormat::UNSIGNED_INT32:                                     cMin = 0.0f;                    cMax = 4294967040.f;    break; // Maximum exactly representable 32-bit integer: (2^24 - 1) * 2^8
                case TextureFormat::HALF_FLOAT:                                         cMin = -1e3f;                   cMax = 1e3f;                    break;
                case TextureFormat::FLOAT:                                                      cMin = -1e5f;                   cMax = 1e5f;                    break;
                case TextureFormat::FLOAT64:                                            cMin = -1e5f;                   cMax = 1e5f;                    break;