1 OpenGL and OpenGL ES 2.0/3.X Conformance Test Instructions
4 This document describes how to build, port, and run the OpenGL and OpenGL ES
5 2.0/3.X conformance tests, and how to verify and submit test results.
7 The Conformance Tests are built on dEQP framework. dEQP documentation is
8 available at http://source.android.com/devices/graphics/testing.html
12 ------------------------
13 - [Test History](#test-history)
14 - [Introduction](#introduction)
15 - [Test Environment Requirements](#test-environment-requirements)
16 - [Configuring and Building the Tests](#configuring-and-building-the-tests)
17 - [Configuration](#configuration)
18 - [Building the Tests](#building-the-tests)
23 - [Common Porting Changes](#common-porting-changes)
24 - [Other Allowable Porting Changes](#other-allowable-porting-changes)
25 - [Running the Tests](#running-the-tests)
26 - [Conformance runs](#conformance-runs)
27 - [Linux and Windows](#linux-and-windows)
28 - [Android](#android-1)
29 - [Running Subsets](#running-subsets)
30 - [Command line options](#command-line-options)
31 - [Understanding the Results](#understanding-the-results)
32 - [Test Logs](#test-logs)
33 - [Debugging Test Failures](#debugging-test-failures)
35 - [Creating a Submission Package](#creating-a-submission-package)
36 - [Submission Update Package](#submission-update-package)
37 - [Passing Criteria](#passing-criteria)
38 - [Troubleshooting](#troubleshooting)
39 - [Crashes early on in the run](#crashes-early-on-in-the-run)
40 - [Build fails](#build-fails)
41 - [Adding new tests](#adding-new-tests)
42 - [Acknowledgments](#acknowledgments)
43 - [Revision History](#revision-history)
46 ------------------------
47 The OpenGL and OpenGL ES Conformance Tests are expanded versions of the
48 OpenGL ES 2.x Conformance Test. Much of the development was done by Symbio, Inc.
49 under a contract with The Khronos Group. drawElements donated a considerable
50 number of new tests and a new execution framework for version 1.1.
51 The tests are built from the same source code base, although some individual
52 feature tests are specific to OpenGL or OpenGL ES and their specification
53 versions, and compilation options differing between OpenGL and OpenGL ES affect
54 how the tests are compiled and executed in some cases.
57 ------------------------
59 This document contains instructions for certifying conformance of implementations
60 of the OpenGL and OpenGL ES APIs. The steps of the process are as follows:
62 1. Configure the conformance tests and port them to your platform.
63 2. Build a test executable and run it against your implementation to produce
65 3. Debug any test failures and modify your implementation as needed until it
67 4. Create a Submission Package containing your final result logs and other
68 documents describing the tested platform.
69 5. Submit the results to the appropriate Review Committee via the
70 Khronos Adopters web page. The Committee will examine your submission and will
71 notify you within thirty days if they find any issues requiring action on your part.
73 This document describes each of these steps in detail. It also provides advice
74 on reproducing, understanding, and debugging test failures, and discusses how
75 to extend or modify the tests and the test framework.
77 The reader is assumed to be a fluent programmer experienced with command line
78 utilities and build tools, such as CMake or Make.
80 Test Environment Requirements
81 ------------------------
83 The conformance tests require a file system. The file system requires support
84 for long file names (i.e. > 8.3 name format). Source files in the conformance
85 tests use mixed case file names. When the `--verbose` option is used, rendered
86 images and test case shaders are copied to the log files. This can lead to quite
87 large log files, up to hundreds of megabytes on disk.
89 Each execution of the conformance test writes a text-format results log to a disk.
90 You will need to include this log as part of your conformance submission package.
92 The conformance test executable can be large. Compiler options and CPU instruction
93 sets can cause substantial variation. The disk space required for the build
94 including all the temporary files can be up to 400MB.
96 The build environment is expected to support C++ with exceptions and
97 the Standard Template Library (STL).
99 Configuring and Building the Tests
100 ------------------------
101 The CTS is built via CMake build system. The requirements for the build are as follows:
102 - CMake 2.8 or newer, 2.8.8 or newer recommended
103 - C++ compiler with STL and exceptions support
104 - Unix: Make + GCC / Clang
105 - Windows: Visual Studio or Windows SDK (available free-of-charge)
106 - Android: Android SDK and NDK for host platform
108 The build is controlled by the file CMakeLists.txt found at the root of
111 If the platform and compiler tools you use are not supported, you may be able to
112 add support for that platform and tools to the build system. If you do this,
113 please submit your changes back to Khronos for inclusion in the official tests
116 Otherwise, if you choose not to use the supplied Makefiles, you must construct
117 an equivalent build system for the chosen development environment(s).
121 The build is configured by using `CMakeLists.txt` files in the build target
122 directory (`targets/`). They specify platform-specific configuration, including
123 include paths and link libraries.
125 The main `CMakeLists.txt` includes the target file based on the `DEQP_TARGET`
126 variable. For example `-DDEQP_TARGET=my_target` will use the target description
127 file `targets/my_target/my_target.cmake`.
129 See the main `CMakeLists.txt` file for the description of the variables that
130 the target file can set.
132 Porting to a new platform includes either creating a new target file, or
133 modifying an existing target description.
135 **NOTE**: All paths, except `TCUTIL_PLATFORM_SRCS` are relative to root source
136 directory. `TCUTIL_PLATFORM_SRCS` is relative to `framework/platform` directory.
138 Following target files are provided with the package:
140 | Name | Description |
141 |:---------|-----------------|
142 |android | Used in Android build. Requires use of suitable toolchain file (see `cmake/` directory) |
143 |default| Checks for presence of GL, ES2, ES3, and EGL libraries and headers in default search paths and configures build accordingly|
144 |null | Null build target |
145 |nullws | NullWS build target |
146 |x11_egl| X11 build for platforms with native EGL support|
147 |x11_glx| X11 build for platforms with native GLX support|
148 |x11_egl_glx| X11 build for platforms with native EGL/GLX support|
150 **Example target file (targets/null/null.cmake):**
152 message("*** Using null context target")
154 set(DEQP_TARGET_NAME "Null")
156 set(TCUTIL_PLATFORM_SRCS
157 null/tcuNullPlatform.cpp
158 null/tcuNullPlatform.hpp
159 null/tcuNullRenderContext.cpp
160 null/tcuNullRenderContext.hpp
161 null/tcuNullContextFactory.cpp
162 null/tcuNullContextFactory.hpp
166 **Common configuration variables and their default values in CMake syntax:**
170 set(DEQP_TARGET_NAME "UNKNOWN")
173 - List of link libraries per API. If no libraries are specified, entry points
174 are loaded at run-time by default for OpenGL ES APIs. EGL always requires link
175 libraries. OpenGL always uses run-time loading.
177 set(DEQP_GLES2_LIBRARIES )
178 set(DEQP_GLES3_LIBRARIES )
179 set(DEQP_GLES31_LIBRARIES )
180 set(DEQP_GLES32_LIBRARIES )
181 set(DEQP_EGL_LIBRARIES )
182 set(DEQP_OPENGL_LIBRARIES )
185 - Generic platform libraries required to link a working OpenGL (ES) Application
186 (e.g. X11 libraries on Unix/X11)
188 set(DEQP_PLATFORM_LIBRARIES )
191 - Libraries / binaries that need to be copied to the build target dir
193 set(DEQP_PLATFORM_COPY_LIBRARIES )
196 - If running on Linux using X11 for creating windows etc., enable this.
198 set(DEQP_USE_X11 OFF)
201 - Embed the test files in the test Before building with this set (if GTF module is present), run these commands:
203 cd external/kc-cts/src/GTF_ES/glsl/GTF
204 perl mergeTestFilesToCSource.pl
207 In your target `.cmake` file add
209 set(DEQP_EMBED_TESTS ON)
210 add_definitions(-DHKEMBEDDEDFILESYSTEM)
213 ### Building the Tests
215 To build the framework, you need first to download sources for zlib, libpng.
217 To download sources, run:
219 python external/fetch_sources.py
221 For OpenGL CTS releases, and OpenGL ES CTS releases prior to opengl-es-cts-3.2.4.0
222 download Khronos Confidential Conformance Test Suite:
224 python external/fetch_kc_cts.py
226 For OpenGL CTS releases, and OpenGL ES CTS releases prior to opengl-es-cts-3.2.4.0
227 the results for the tests included in this suite must be included in a
228 conformance submission.
230 **NOTE**: You need to be a Khronos Adopter and have an active account
231 at [Khronos Gitlab](https://gitlab.khronos.org/) to be able to download
232 Khronos Confidential CTS.
233 It is possible to run and build the CTS without the Khronos Confidential CTS.
234 For OpenGL CTS releases, and OpenGL ES CTS releases prior to opengl-es-cts-3.2.4.0
235 Khronos Confidential CTS is mandatory if you plan to make a
236 conformance submission (see [Creating a Submission Package](#creating-a-submission-package)).
237 For opengl-es-cts-3.2.4.0 and later OpenGL ES CTS releases Khronos Confidential CTS
238 results must not be included in a submission package.
241 With CMake out-of-source builds are always recommended. Create a build directory
242 of your choosing, and in that directory generate Makefiles or IDE project
248 - Visual Studio (2010 or newer recommended) or Windows SDK
249 - CMake 2.8.x Windows native version (i.e. not Cygwin version)
250 - For GL/ES2/ES3.x tests: OpengGL, OpenGL ES 2 or ES 3.x libraries and headers
252 To choose the backend build system for CMake, choose one of the following Generator Names for the
253 command line examples in the next steps:
254 - VS2010: "Visual Studio 10"
255 - VS2012: "Visual Studio 11"
256 - NMake (must be run in VS or SDK command prompt): "NMake Makefiles"
258 Building GL, ES2, or ES3.x conformance tests:
260 cmake <path to openglcts> -DDEQP_TARGET=default -G"<Generator Name>"
263 Khronos Confidential CTS doesn't support run-time selection of API context.
264 If you intend to run it you need to additionally supply `GLCTS_GTF_TARGET`
265 option to you cmake command, e.g.:
267 cmake <path to openglcts> -DDEQP_TARGET=default -DGLCTS_GTF_TARGET=<target> -G"<Generator Name>"
269 Available `<target>`s are `gles2`, `gles3`, `gles31`, `gles32`, and `gl`.
270 The default `<target>` is `gles32`.
272 It's also possible to build `GL-CTS.sln` in Visual Studio instead of running
273 the `cmake --build .` command.
275 **NOTE**: Do not create the build directory under the source directory
276 (i.e anywhere under `<path to openglcts>`) on Windows, since it causes
277 random build failures when copying data files around.
279 **NOTE**: You can use the CMake for Windows GUI to do configuration and project
282 **NOTE**: If using cygwin, you must install and ensure you use the Windows
283 version of cmake. The cygwin vesion does not contain the Visual Studio
284 generators. Here is a shell function you can put in your cygwin `.bash_profile`
285 to use it easily. With this you can simply type `wcmake` to run the Windows version.
289 (TMP=$tmp TEMP=$temp; unset tmp; unset temp; "C:/Program Files (x86)/CMake 2.8/bin/cmake" "$@")
296 - Standard build utilities (make, gcc, etc.)
298 - Necessary API libraries (OpenGL, GLES, EGL depending on configuration)
300 Building ES2 or ES3.x conformance tests:
302 cmake <path to openglcts> -DDEQP_TARGET=null -DGLCTS_GTF_TARGET=gles32
305 Building OpenGL conformance tests:
307 cmake <path to openglcts> -DDEQP_TARGET=null -DGLCTS_GTF_TARGET=gl
310 Khronos Confidential CTS doesn't support run-time selection of API context.
311 If you intend to run it then the `GLCTS_GTF_TARGET` option is necessary.
313 Available values for `GLCTS_GTF_TARGET` are `gles2`, `gles3`, `gles31`, `gles32`, and `gl`.
314 The default value is `gles32`.
316 CMake chooses to generate Makefiles by default. Other generators can be used
317 as well. See CMake help for more details.
321 The conformance tests come with native Android support. The following packages
322 are needed in order to build an Android binary:
325 - Android SDK with API 24 packages and tools installed
328 An Android binary (for ES 3.2) can be built using command:
330 python scripts/android/build_apk.py --target=openglcts --sdk <path to Android SDK> --ndk <path to Android NDK>
332 If Khronos Confidential CTS is present then the script will set `GLCTS_GTF_TARGET`
333 to `gles32` by default.
334 It is possible to specify a different `GLCTS_GTF_TARGET` target by invoking the script
335 with the `--kc-cts-target` option, e.g.:
337 python scripts/android/build_apk.py --target=openglcts --kc-cts-target=gles31 --sdk <path to Android SDK> --ndk <path to Android NDK>
339 Available values for `--kc-cts-target` are `gles32`, `gles31`, `gles3`, `gles2` and `gl`.
341 The package can be installed by either running:
343 python scripts/android/install_apk.py --target=openglcts
345 By default the CTS package will contain libdeqp.so built for `armeabi-v7a`, `arm64-v8a`,
346 `x86`, and `x86_64` ABIs, but that can be changed with `--abis` command line option.
348 To pick which ABI to use at install time, following commands must be used
351 adb install --abi <ABI name> <build root>/Khronos-CTS.apk /data/local/tmp/Khronos-CTS.apk
354 ------------------------
355 The Conformance Tests have been designed to be relatively platform-, OS-, and
356 compiler-independent. Adopters are responsible for final changes needed to allow
357 the Test to run on the platform they wish to
358 certify as conformant.
360 ### Common Porting Changes
362 Porting the dEQP framework requires implementation of either `glu::Platform` or,
363 on platforms supporting EGL, the `tcu::EglPlatform` interface. The porting layer
364 API is described in detail in following files:
366 framework/common/tcuPlatform.hpp
367 framework/opengl/gluPlatform.hpp
368 framework/egl/egluPlatform.hpp
369 framework/platform/tcuMain.cpp
371 This version of the dEQP framework includes ports for Windows (both EGL and WGL),
372 X11 (EGL and XGL), and Android.
374 Base portability libraries in `framework/delibs` seldom need changes. However,
375 introducing support for a new compiler or a new processor family may require
376 some changes to correctly detect and parameterize the environment.
378 Porting typically involves three types of changes:
379 1. Changes to the make system used to generate the test executable.
380 2. Changes needed to adapt the test executable to the operating system used on the platform.
381 3. Changes to the platform specific GL and EGL header files.
383 Changes should normally be confined to build files (CMake or Python) or source
384 files (.c, .h, .cpp, and .h files) in the following directories or their
386 - `framework/platform`
389 If you find that you must change other source (.c, .cpp, .h, or .hpp) files,
390 you will need to file a waiver as described below.
392 Note that the conformance tests assume that the implementation supports EGL.
393 However EGL is not required for OpenGL or OpenGL ES conformance.
395 Most of the tests require at least 256x256 pixels resolution in order to run properly
396 and produce stable results. It is, therefore, important to ensure that a port to a
397 new platform can support surfaces that fulfill width and height requirements.
399 ### Other Allowable Porting Changes
401 Other than changes needed for porting, the only changes that are permitted are
402 changes to fix bugs in the conformance test. A bug in the conformance test is
403 a behavior which causes clearly incorrect execution (e.g., hanging, crashing,
404 or memory corruption), OR which requires behavior which contradicts or exceeds
405 the requirements of the relevant OpenGL or OpenGL ES Specification. Changes
406 required to address either of these issues typically require [waivers](#waivers).
409 ------------------------
410 All the following commands need to be run in the CTS build directory. If you
411 need to move the binaries from the build directory, remember to copy the
412 data directories named `gl_cts`, `gles2`, `gles3`, and `gles31` and its subdirectories
413 from the build directory to the test target in the same relative locations.
415 If the build instructions have been followed as-is, the correct path is:
417 cd <builddir>/external/openglcts/modules
420 A conformance run can be launched either by running the `cts-runner` binary with
421 appropriate options on Linux/Windows or by running an Android application.
423 ### Linux and Windows
424 Conformance run for OpenGL ES 3.2 on Windows:
426 Debug/cts-runner.exe --type=es32
427 [For ES 3.1 use --type=es31; ES 3.0 use --type=es3; for ES 2.0, use --type=es2]
429 Conformance run for OpenGL 3.0 - 4.4 on Windows:
431 Debug/cts-runner.exe --type=glxy
432 [x and y are the major and minor specifiction versions]
434 Full list of parameters for the `cts-runner` binary:
436 --type=[esN[M]|glNM] Conformance test run type. Choose from
437 ES: es2, es3, es31, es32
438 GL: gl30, gl31, gl32, gl33, gl40, gl41, gl42, gl43, gl44, gl45
439 --logdir=[path] Destination directory for log files
440 --summary Print summary without running the tests
441 --verbose Print out and log more information
444 The conformance run will create one or more `.qpa` files per tested config, a
445 summary `.qpa` file containing run results and a summary `.xml` file containing
446 command line options for each run, all of which should be included in your
447 conformance submission package. The final verdict will be printed out at
450 Sometimes it is useful to know the command line options used for the conformance
451 before the run completed. Full conformance run configuration is written
452 to `cts-run-summary.xml` and this file can be generated by adding `--summary`
455 By default the `cts-runner` does not include result images or shaders used in
456 the logs. Adding parameter `--verbose` will cause them to be included in
457 the logs. Images will be embedded as PNG data into the`.qpa` log files.
458 See Section [Test Logs](#test-logs) for instructions on how to view the images.
460 To direct logs to a directory, add `--logdir=[path]` parameter.
462 **NOTE**: Due to the lack of support for run-time selection of API context in the
463 Khronos Confidential CTS, a conformance run may fail if it is executed for an API
464 version that doesn't match the `GLCTS_GTF_TARGET` value used during the build step.
468 Once the CTS binary is built and installed on the device, a new application
469 called `ES3.2 CTS`, `ES3.1 CTS`, `ES3 CTS`, `ES2 CTS`, or `GL4.5 CTS` (depending
470 on the test version you built) should appear in the launcher. Conformance test
471 runs can be done by launching the applications.
473 Alternatively it is possible to start a conformance run from the command line,
474 for example to launch a GLES 3.2 conformance run use:
476 am start -n org.khronos.gl_cts/org.khronos.cts.ES32Activity -e logdir "/sdcard/logs"
478 For GLES 2.0, GLES 3.0, GLES 3.1, or GL 4.5 conformance runs, substitute the following
479 activity name (respectively) ES2Activity, ES3Activity, ES31Activity, or GL45Activity.
481 Test logs will be written to `/sdcard` by default. The log path can be
482 customized by supplying a `logdir` string extra in launch intent. Verbose mode
483 can be enabled by supplying a `verbose` = `"true"` string extra. See
484 the following example:
486 am start -n org.khronos.gl_cts/org.khronos.cts.ES32Activity -e logdir "/sdcard/logs" -e verbose "true"
488 Conformance run configuration can be generated by supplying a `summary` = `"true"`
489 string extra. See the following example:
491 am start -n org.khronos.gl_cts/org.khronos.cts.ES32Activity -e logdir "/sdcard/logs" -e summary "true"
493 **NOTE**: Supplying a `summary` = `"true"` string extra will result in the `cts-run-summary.xml` file
494 being written out but no tests will be executed.
496 Individual tests can be launched as well by targeting
497 `org.khronos.gl_cts/android.app.NativeActivity` activity. Command line
498 arguments must be supplied in a `cmdLine` string extra. See following example:
500 am start -n org.khronos.gl_cts/android.app.NativeActivity -e cmdLine "cts --deqp-case=KHR-GLES32.info.version --deqp-gl-config-id=1 --deqp-log-filename=/sdcard/ES32-egl-config-1.qpa --deqp-surface-width=128 --deqp-surface-height=128"
502 In addition to the detailed `*.qpa` output files, the Android port of the CTS
503 logs a summary of the test run, including the pass/fail status of each test.
504 This summary can be viewed using the Android *logcat* utility.
506 See Section [Running Subsets](#running-subsets) above for details on command
511 Run shader compiler loop test cases from the OpenGL ES 3.0 CTS using EGL config with ID 3:
513 Debug/glcts.exe --deqp-case=KHR-GLES3.shaders.loops.* --deqp-gl-config-id=3
515 Note that the GL context version is determined by the case name. `KHR-GLES3` in
516 the example above selects OpenGL ES 3.0. The command to run the same test
517 against OpenGL version 4.1 is:
519 Debug/glcts.exe --deqp-case=GL41-CTS.shaders.loops.* --deqp-gl-config-id=3
521 To list available test cases (writes out `*-cases.txt` files per module), run:
523 Debug/glcts.exe --deqp-runmode=txt-caselist
525 The type of the run for cts-runner chooses a specific list of test cases to
526 be run. The selected tests can be checked from the summary logs. To run
527 the same tests, just give equivalent test selection parameters to the `glcts`.
529 #### Command line options
531 Full list of parameters for the `glcts` binary:
536 -n, --deqp-case=<value>
537 Test case(s) to run, supports wildcards (e.g. dEQP-GLES2.info.*)
539 --deqp-caselist=<value>
540 Case list to run in trie format (e.g. {dEQP-GLES2{info{version,renderer}}})
542 --deqp-caselist-file=<value>
543 Read case list (in trie format) from given file
545 --deqp-stdin-caselist
546 Read case list (in trie format) from stdin
548 --deqp-log-filename=<value>
549 Write test results to given file
550 default: 'TestResults.qpa'
552 --deqp-runmode=[execute|xml-caselist|txt-caselist|stdout-caselist]
553 Execute tests, or write list of test cases into a file
556 --deqp-caselist-export-file=<value>
557 Set the target file name pattern for caselist export
558 default: '${packageName}-cases.${typeExtension}'
560 --deqp-watchdog=[enable|disable]
564 --deqp-crashhandler=[enable|disable]
565 Enable crash handling
568 --deqp-base-seed=<value>
569 Base seed for test cases that use randomization
572 --deqp-test-iteration-count=<value>
573 Iteration count for cases that support variable number of iterations
576 --deqp-visibility=[windowed|fullscreen|hidden]
577 Default test window visibility
580 --deqp-surface-width=<value>
581 Use given surface width if possible
584 --deqp-surface-height=<value>
585 Use given surface height if possible
588 --deqp-surface-type=[window|pixmap|pbuffer|fbo]
589 Use given surface type
592 --deqp-screen-rotation=[unspecified|0|90|180|270]
593 Screen rotation for platforms that support it
596 --deqp-gl-context-type=<value>
597 OpenGL context type for platforms that support multiple
599 --deqp-gl-config-id=<value>
600 OpenGL (ES) render config ID (EGL config id on EGL platforms)
603 --deqp-gl-config-name=<value>
604 Symbolic OpenGL (ES) render config name
606 --deqp-gl-context-flags=<value>
607 OpenGL context flags (comma-separated, supports debug and robust)
609 --deqp-cl-platform-id=<value>
610 Execute tests on given OpenCL platform (IDs start from 1)
613 --deqp-cl-device-ids=<value>
614 Execute tests on given CL devices (comma-separated, IDs start from 1)
617 --deqp-cl-build-options=<value>
618 Extra build options for OpenCL compiler
620 --deqp-egl-display-type=<value>
621 EGL native display type
623 --deqp-egl-window-type=<value>
624 EGL native window type
626 --deqp-egl-pixmap-type=<value>
627 EGL native pixmap type
629 --deqp-log-images=[enable|disable]
630 Enable or disable logging of result images
633 --deqp-log-shaders=[enable|disable]
634 Enable or disable logging of shaders
637 --deqp-test-oom=[enable|disable]
638 Run tests that exhaust memory on purpose
641 --deqp-egl-config-id=<value>
642 Legacy name for --deqp-gl-config-id
645 --deqp-egl-config-name=<value>
646 Legacy name for --deqp-gl-config-name
649 ### Understanding the Results
651 At the end of a completed test run, a file called `cts-run-summary.xml` is
652 generated. It will contain summaries per configuration and the full command
653 lines for the `glcts` application
654 (See Section [Running Subsets](#running-subsets)) for debugging purposes.
655 Additionally, a summary string similar to one below is printed:
657 4/4 sessions passed, conformance test PASSED
660 If the run fails, the message will say `FAILED` instead of `PASSED`. Under
661 Linux or Windows, this string is printed to stdout if available. Under Android,
662 it is emitted to the Android logging system for access via *logcat*.
664 Each test case will be logged into the `.qpa` files in XML. Below is a minimal
665 example of a test case log. The Result element contains the final verdict in
666 the `StatusCode` attribute. Passing cases will have `Pass` and failing cases
667 `Fail`. Other results such as `QualityWarning`, `CompatibilityWarning`,
668 `NotSupported` or `ResourceError` are possible. Only `Fail` status will count
669 as failure for conformance purposes.
671 <TestCaseResult Version="0.3.2" CasePath="ES2-CTS.info.vendor" CaseType="SelfValidate">
672 <Text>Vendor A</Text>
673 <Result StatusCode="Pass">Pass</Result>
677 If the failure count is zero for all config sequences, the implementation
678 passes the test. Note that in addition to a successful test result,
679 a Submission Package must satisfy the conditions specified below under
680 [Passing Criteria](#passing-criteria) in order to achieve conformance certification.
684 The CTS writes test logs in XML encapsulated in a simple plain-text container
685 format. Each tested configuration listed in `cts-run-summary.xml`
687 To analyse and process the log files, run the following scripts
688 - `external/openglcts/scripts/verify_submission.py`: Script that verifies logs based on `cts-run-summary.xml` file.
689 - `scripts/log/log_to_csv.py`: This utility converts `.qpa` log into CSV format. This is
690 useful for importing results into other systems.
691 - `scripts/log/log_to_xml.py`: Converts `.qpa` into well-formed XML document. The document
692 can be then viewed in browser using the testlog.{xsl,css} files.
694 Some browsers, like Chrome, limit local file access. In such case, the files
695 must be accessed over HTTP. Python comes with a simple HTTP server suitable
696 for the purpose. Run `python -m SimpleHTTPServer` in the directory containing
697 the generated XML files and point the browser to `127.0.0.1:8000`.
699 Parser for the `.qpa` log file format in python is provided in
700 `scripts/log/log_parser.py`.
702 Python scripts require python 2.7 or newer in 2.x series. They are not
703 compatible with python 3.x.
705 Debugging Test Failures
706 ------------------------
707 The best first step is to run the failing test cases via `glcts` executable to
708 get the more verbose logs. Use, for example, the `log_to_xml.py` script
709 detailed in Section [Test Logs](#test-logs), to view the generated logs.
710 If the visual inspection of the logs does not give sufficient hints on the
711 nature of the issue, inspecting the test code and stepping through it in
712 debugger should help.
715 ------------------------
716 The procedure for requesting a waiver is to report the issue by filing a bug
717 report in the Gitlab VK GL CTS project
718 (https://gitlab.khronos.org/Tracker/vk-gl-cts). When you create your submission
719 package, include references to the waivers as described in the adopters' agreement.
720 [Fully-qualified links](https://en.wikipedia.org/wiki/Fully_qualified_domain_name)
721 to bug reports are highly recommended.
722 Including as much information as possible in your bug report will ensure the issue
723 can be progressed as speedily as possible. Such bug report must
724 include a link to suggested file changes. Issues must be labeled `Waiver` and `OpenGL-ES`
725 (for OpenGL ES submissions) or `Waiver` and `OpenGL` (for OpenGL submissions) and
726 identify the CTS release tag and affected tests.
728 Creating a Submission Package
729 ------------------------
730 Please see the [Creating a Submission Package page](https://github.com/KhronosGroup/VK-GL-CTS/wiki/Creating-a-OpenGL-and-OpenGL-ES-Submission-Package).
732 Submission Update Package
733 ------------------------
734 Please see the [Submission Update Package page](https://github.com/KhronosGroup/VK-GL-CTS/wiki/Submission-Update-Package).
737 ------------------------
738 Please see the [Conformance Submission Passing Criteria page](https://github.com/KhronosGroup/VK-GL-CTS/wiki/OpenGL-and-OpenGL-ES-Conformance-Submission-Passing-Criteria).
741 ------------------------
742 ### Crashes early on in the run
743 If using run-time entry point loading, it is possible that not all required
744 entry points are available. This will result in `NULL` pointer dereferencing.
747 First try re-running the build. If that does not help and you have used the
748 same build directory with different version of the CTS, remove the build
749 directory and run the CMake again.
752 ------------------------
754 See the [Contribution Guide](CONTRIBUTING.md)
757 ------------------------
758 The Khronos Group gratefully acknowledges the support of drawElements Oy,
759 who donated a large number of GLSL tests and a new test framework and build system.
761 The Khronos Group also gratefully acknowledges the support of 3DLabs Inc.,
762 who gave permission to use the 3DLabs Graphics Test Framework (GTF).
764 The first internal version of the test was created by Bruno Schwander of
765 Hooked Wireless, under a development contract with the Khronos Group.
767 Symbio added tests specific to OpenGL and OpenGL ES 3.0.
769 drawElements added their donated language tests and build system.
771 The CTS results from these efforts, together with additional hard work by
772 volunteers from the OpenGL ES Working Group, the OpenGL ARB Working Group,
773 and their member companies, including:
775 - Sumit Agarwal, Imagination Technologies
777 - Oleksiy Avramchenko, Sony
778 - Anthony Berent, ARM
779 - Joseph Blankenship, AMD
781 - Pierre Boudier, AMD
782 - Benji Bowman, Imagination Technologies
784 - David Cairns, Apple
785 - Mark Callow, ArtSpark
786 - Antoine Chauveau, NVIDIA
787 - Aske Simon Christensen, ARM
789 - Mathieu Comeau, QNX
790 - Graham Connor, Imagination Technologies
791 - Slawomir Cygan, Intel
792 - Piotr Czubak, Intel
793 - Piers Daniell, NVIDIA
794 - Matthias Dejaegher, ZiiLabs
796 - David Donohoe, Movidius
799 - Bryan Eyler, NVIDIA
800 - Erik Faye-Lund, ARM
801 - Nicholas FitzRoy-Dale, Broadcom
802 - Michael Frydrych, NVIDIA
803 - Toshiki Fujimori, Takumi
804 - David Garcia, Qualcomm
805 - Frido Garritsen, Vivante
806 - Klaus Gerlicher, NVIDIA
807 - Slawomir Grajewski, Intel
808 - Jonas Gustavsson, Sony
809 - Nick Haemel, NVIDIA
810 - Matthew Harrison, Imagination Technologies
811 - Pyry Haulos, drawElements
812 - Jim Hauxwell, Broadcom
813 - Valtteri Heikkil, Symbio
815 - Mathias Heyer, NVIDIA
816 - Atsuko Hirose, Fujitsu
817 - Ari Hirvonen, NVIDIA
819 - Jaakko Huovinen, Nokia
820 - James Jones, Imagination Technologies
821 - Norbert Juffa, NVIDIA
822 - Jordan Justen, Intel
823 - Sandeep Kakarlapudi, ARM
824 - Anssi Kalliolahti, NVIDIA
825 - Philip Kamenarsky, NVIDIA
826 - Krzysztof Kaminski, Intel
827 - Daniel Kartch, NVIDIA
829 - Jon Kennedy, 3DLabs
831 - Daniel Koch, NVIDIA
832 - Benjamin Kohler-Crowe, NVIDIA
833 - Georg Kolling, Imagination Technologies
835 - Boguslaw Kowalik, Intel
836 - Aleksandra Krstic, Qualcomm
837 - Karol Kurach, NVIDIA
839 - Sami Kyostila, Google
840 - Teemu Laakso, Symbio
841 - Antoine Labour, Sony
842 - Alexandre Laurent, Imagination Technologies
844 - Graeme Leese, Broadcom
845 - I-Gene Leong, Intel
846 - Radoslava Leseva, Imagination Technologies
848 - Fred Liao, MediaTek
849 - Bill Licea-Kane, Qualcomm
850 - Benj Lipchak, Apple
851 - Wayne Lister, Imagination Technologies
854 - Zhifang Long, Marvell
855 - Toni Lönnberg, AMD
857 - Christer Lunde, ARM
859 - Daniel Mahashin, NVIDIA
860 - Rob Matthesen, NVIDIA
861 - Tom McReynolds, NVIDIA (CTS TSG Chair, ES 1.1)
863 - Assif Mirza, Imagination Technologies
865 - Kazuhiro Mochizuki, Fujitsu
866 - Affie Munshi, Apple
867 - Yeshwant Muthusamy, Samsung
868 - Mirela Nicolescu, Broadcom
869 - Glenn Nissen, Broadcom
870 - Michael O'Hara, AMD
871 - Eisaku Ohbuchi, DMP
873 - Tapani Palli, Intel
876 - Adrian Peirson, ARM
877 - Russell Pflughaupt, NVIDIA
879 - Tero Pihlajakoski, Nokia
880 - Peter Pipkorn, NVIDIA
881 - Acorn Pooley, NVIDIA
882 - Guillaume Portier, ArtSpark
883 - Greg Prisament, Lychee Software
884 - Jonathan Putsman, Imagination Technologies
886 - Tarik Rahman, CodePlay
887 - Kalle Raita, drawElements
889 - Manjunatha Ramachandra
890 - John Recker, NVIDIA
891 - Maurice Ribble, Qualcomm (CTS TSG Chair, ES 2.0)
892 - James Riordon, Khronos
893 - Lane Roberts, Samsung
894 - Ian Romanick, Intel
896 - Kenneth Russell, Google
897 - Matteo Salardi, Imagination Technologies
898 - Jeremy Sandmel, Apple
899 - Shusaku Sawato, DMP
900 - Chris Scholtes, Fujitsu
901 - Mathias Schott, NVIDIA
902 - Bruno Schwander, Hooked Wireless
903 - Graham Sellers, AMD
904 - Shereef Shehata, Texas Instruments
905 - Benjamin Shen, Vivante
906 - Robert Simpson, Qualcomm
907 - Stuart Smith, Imagination Technologies
908 - Janusz Sobczak, Mobica
909 - Jacob Strom, Ericsson
910 - Timo Suoranta, Broadcom
911 - Jan Svarovsky, Ideaworks3D
913 - Payal Talati, Imagination Technologies
914 - Gregg Tavares, Google
915 - Ross Thompson, NVIDIA
916 - Jeremy Thorne, Broadcom
917 - Jani Tikkanen, Symbio
918 - Antti Tirronen, Qualcomm (CTS TSG Chair, ES 3.0/3.1)
919 - Robert Tray, NVIDIA
921 - Eben Upton, Broadcom
922 - Jani Vaarala, Nokia
923 - Dmitriy Vasilev, NVIDIA
924 - Chad Versace, Intel
925 - Holger Waechtler, Broadcom
927 - Jun Wang, Imagination Technologies
928 - Yuan Wang, Imagination Technologies
930 - Ewa Wisniewska, Mobica
931 - Dominik Witczak, Mobica
932 - Oliver Wohlmuth, Fujitsu
933 - Yanjun Zhang, Vivante
934 - Lefan Zhong, Vivante
936 - Marek Zylak, NVIDIA
937 - Iliyan Dinev, Imagination Technologies
938 - James Glanville, Imagination Technologies
940 - Alexander Galazin, ARM
941 - Riccardo Capra, ARM
942 - Lars-Ivar Simonsen, ARM
946 ------------------------
949 Initial version cloned from `ES2_Readme`, plus feedback from Mark Callow.
953 Modified to incorporate feedback in bug 8534.
957 Added details for OpenGL Conformance.
959 - 0.4 - Jon Leech 2012/10/31
961 Add configuration & build section, and table of contents
963 - 0.5 - Jon Leech 2012/10/31
965 Fix typos noted by Mark Callow in bug 8534.
967 - 0.6 - Jon Leech 2012/11/13
969 Discuss automatic version selection and document support for OpenGL 3.3-4.3.
971 - 0.7 - Jon Leech 2012/11/14
973 Minor cleanup for GL version numbers per Bug 8534 comment #41.
975 - 0.8 - Tom Olson 2013/1/25
977 Updated GL status in preparation for ES 3.0 release, removed display
978 parameters from product description, and removed mention of sample submission.
980 - 0.9 - Jon Leech 2013/07/17
982 Restore GL-specific details in preparation for initial GL CTS release.
984 - 1.0 - Jon Leech 2013/07/17
986 Change references to Visual Studio 11 to Visual Studio 2012 per bug 9862.
987 Reset change tracking to reduce clutter.
989 - 1.1 - Kalle Raita 2013/10/30
991 Updated documentation after the integration of the drawElements framework and
994 - 1.2 - Kalle Raita 2013/12/03
996 Removed TODOs, added some notes on further development, and notes on file
997 dependencies. Exact list of directory sub-trees that can be modified during porting.
999 - 1.3 - Tom Olson 2014/05/27
1001 Updates for ES CTS 3.1.1.0 . Added Passing Criteria, updated examples to
1002 include 3.1 versioning, and updated Acknowledgements.
1004 - 1.4 - Alexander Galazin 2016/05/12
1006 Updates for ES CTS 3.2.1.0.
1008 - 2.0 - Alexander Galazin 2016/09/23
1010 Moved the contents to README.md.
1011 Updated to reflect new CTS structure and build instructions.
1013 - 2.1 - Alexander Galazin 2016/12/15
1015 Updates in preparation for the new release.
1016 Document restructuring, more detailed process of creating a submission package.
1017 Incorporated OpenGL/CTS issue 39 and 40 in the Passing Criteria.