Add support for metamorphic testing
authorHugues Evrard <h.evrard@imperial.ac.uk>
Wed, 19 Apr 2017 10:03:12 +0000 (11:03 +0100)
committerPyry Haulos <phaulos@google.com>
Tue, 16 May 2017 16:33:29 +0000 (09:33 -0700)
commit354195abe304cbf9cdc6a83f89d9e059edc4cead
treebc4abcea4cac424fbc8f9268a7a46b0c25218b15
parentab92cdbb586cb447900c47c6f538d4c5ffb82c4a
Add support for metamorphic testing

Add a new test case which compare the renderings of two different yet
semantically equivalent fragment shaders. A couple of shadersets are
also added as associated data, each contains one recipient fragment
shader which is the reference, and a few variants which are semantically
equivalent to the recipient. Each variant included here has proven to
trigger some issue on an Android device, see comment in the variant
source code.

Besides comparing the renderings of both recipient and variant shaders,
the test case also checks that each shader leads to a deterministic
rendering, by performing rendering twice and doing a pixel-per-pixel
comparison on the resulting image. When comparing the recipient and
variant renderings, a slight difference is tolerated to cope with
variations due to floating point arithmetics. Moreover, a sanity check
function may be used in order to verify that the recipient image is
relevant, and thus avoid a e.g. full black image rendering to pass the
test.

Change-Id: Id6cbbe7459864c6fb3acf72faa393ee0f4d8315d
18 files changed:
AndroidGen.mk
android/cts/master/gles3-master.txt
android/package/Android.mk
external/graphicsfuzz/data/gles3/graphicsfuzz/LICENSE [new file with mode: 0644]
external/graphicsfuzz/data/gles3/graphicsfuzz/bubblesort_flag/recipient.frag [new file with mode: 0755]
external/graphicsfuzz/data/gles3/graphicsfuzz/bubblesort_flag/variant_1.frag [new file with mode: 0755]
external/graphicsfuzz/data/gles3/graphicsfuzz/bubblesort_flag/variant_2.frag [new file with mode: 0755]
external/graphicsfuzz/data/gles3/graphicsfuzz/synthetic/recipient.frag [new file with mode: 0755]
external/graphicsfuzz/data/gles3/graphicsfuzz/synthetic/variant_1.frag [new file with mode: 0755]
external/graphicsfuzz/data/gles3/graphicsfuzz/synthetic/variant_2.frag [new file with mode: 0755]
external/graphicsfuzz/data/gles3/graphicsfuzz/synthetic/variant_3.frag [new file with mode: 0755]
external/graphicsfuzz/data/gles3/graphicsfuzz/synthetic/variant_4.frag [new file with mode: 0755]
external/graphicsfuzz/data/gles3/graphicsfuzz/vertexShader.glsl [new file with mode: 0644]
modules/gles3/CMakeLists.txt
modules/gles3/functional/CMakeLists.txt
modules/gles3/functional/es3fFunctionalTests.cpp
modules/gles3/functional/es3fShaderMetamorphicTests.cpp [new file with mode: 0644]
modules/gles3/functional/es3fShaderMetamorphicTests.hpp [new file with mode: 0644]