Vulkan CTS README ================= This document describes how to build and run Vulkan Conformance Test suite. Vulkan CTS is built on dEQP framework. dEQP documentation is available at http://source.android.com/devices/graphics/testing.html Requirements ------------ Common: * Git (for checking out sources) * Python 2.7.x (all recent versions in 2.x should work, 3.x is not supported) * CMake 2.8 or newer Win32: * Visual Studio 2013 or newer (glslang uses several C++11 features) Linux: * Standard toolchain (make, gcc/clang) Android: * Android NDK r10e * Android SDK with following packages: + SDK Tools + SDK Platform-tools + SDK Build-tools + API 22 * Apache Ant * Windows: either NMake or JOM in PATH Building CTS ------------ To build dEQP, you need first to download sources for zlib, libpng, glslang, and spirv-tools. To download sources, run: $ python external/fetch_sources.py You may need to re-run fetch_sources.py to update to the latest glslang and spirv-tools revisions occasionally. NOTE: glslang integration is not yet available on Android due to a toolchain bug, so pre-compiled SPIR-V binaries must be used. See instructions below. With CMake out-of-source builds are always recommended. Create a build directory of your choosing, and in that directory generate Makefiles or IDE project using cmake. Windows x86-32: > cmake -G"Visual Studio 12" > start dEQP-Core-default.sln Windows x86-64: > cmake -G"Visual Studio 12 Win64" > start dEQP-Core-default.sln Linux 32-bit Debug: $ cmake -DCMAKE_BUILD_TYPE=Debug -DCMAKE_C_FLAGS=-m32 -DCMAKE_CXX_FLAGS=-m32 $ make -j Release build can be done by using -DCMAKE_BUILD_TYPE=Release Linux 64-bit Debug: $ cmake -DCMAKE_BUILD_TYPE=Debug -DCMAKE_C_FLAGS=-m64 -DCMAKE_CXX_FLAGS=-m64 $ make -j Android: Following command will build CTS into android/package/bin/dEQP-debug.apk. $ python android/scripts/build.py The package can be installed by either running: $ python android/scripts/install.py By default the CTS package will contain libdeqp.so built for armeabi-v7a, arm64-v8a, and x86 ABIs, but that can be changed in android/scripts/common.py script. To pick which ABI to use at install time, following commands must be used instead: $ adb install --abi android/package/bin/dEQP-debug.apk /data/local/tmp/dEQP-debug.apk Building Mustpass ----------------- Current mustpass is checked into repository and can be found at: external/vulkancts/mustpass/1.0.0/vk-default.txt Vulkan CTS mustpass can be re-generated by running: $ python /external/vulkancts/build_mustpass.py Pre-compiling SPIR-V binaries ----------------------------- For distribution, and platforms that don't support GLSL to SPIR-V compilation, SPIR-V binaries must be pre-built with following command: $ python external/vulkancts/build_spirv_binaries.py Binaries will be written to external/vulkancts/data/vulkan/prebuilt/. Test modules (or in case of Android, the APK) must be re-built after building SPIR-V programs in order for the binaries to be available. Running CTS ----------- Following command line options MUST be used when running CTS: --deqp-caselist-file=/external/vulkancts/mustpass/1.0.0/vk-default.txt --deqp-log-images=disable --deqp-log-shader-sources=disable In addition on multi-device systems the device for which conformance is claimed can be selected with: --deqp-vk-device-id= No other command line options are allowed. Win32: > cd /external/vulkancts/modules/vulkan > Debug/deqp-vk.exe --deqp-caselist-file=... Test log will be written into TestResults.qpa Linux: $ cd /external/vulkancts/modules/vulkan $ ./deqp-vk --deqp-vk-caselist-file=... Android: $ adb push /external/vulkancts/mustpass/1.0.0/vk-default.txt /sdcard/vk-default.txt $ adb shell In device shell: $ am start -n com.drawelements.deqp/android.app.NativeActivity -e cmdLine "deqp --deqp-caselist-file=/sdcard/vk-default.txt --deqp-log-images=disable --deqp-log-shader-sources=disable --deqp-log-filename=/sdcard/TestResults.qpa" Process can be followed by running: $ adb logcat -s dEQP Test log will be written into /sdcard/TestResults.qpa Conformance Submission Package Requirements ------------------------------------------- The conformance submission package must contain the following: 1) Full test logs (TestResults.qpa) from CTS runs against all driver builds 2) Result of "git status" and "git log" from CTS source directory 3) Any patches used on top of release tag 4) Conformance statement Test logs (1) should be named /TestResults-.qpa, for example TestResults-armeabi-v7a.qpa. On platforms where multiple different driver builds (for example 64-bit and 32-bit) are present, CTS logs must be provided for each driver build. The CTS build must always be done from clean git repository that doesn't have any uncommitted changes. Thus it is necessary to run and capture output of "git status" and "git log" (2) in the source directory: git status > /git-status.txt git log ..HEAD > /git-log.txt Any changes made to CTS must be committed to the local repository, and provided as part of the submission package (3). This can be done by running: git format-patch -o ..HEAD In general, bugfixes and changes to platform-specific code (mostly under framework/platform) are allowed. The commit message for each change must include a clear description of the change and why it is necessary. Non-porting related changes must be accompanied by a waiver (see below). Note: When cherry-picking patches on top of release tag, please use "git cherry-pick -x" to include original commit hash in the commit message. Conformance statement (4) must be included in a file called STATEMENT- and must contain following: CONFORM_VERSION: PRODUCT: CPU: OS: Note that product/cpu/os information is also captured in dEQP-VK.info.* tests if vk::Platform::describePlatform() is implemented. The actual submission package consists of the above set of files which must be bundled into a gzipped tar file named VK10_<_info>.tgz. is the name of the Adopting member company, or some recognizable abbreviation. The <_info> field is optional. It may be used to uniquely identify a submission by OS, platform, date, or other criteria when making multiple submissions. One way to create a suiteable gzipped tar file is to execute the command: $ tar -cvzf -C . where is the directory containing the files from (1)-(4) from above. A submission package must contain all of the files listed above, and only those files. Waivers ------- The process for requesting a waiver is to report the issue by filing a bug report in the Gitlab VulkanCTS project (TODO Github?). When creating the submission package, include references to the waiver in the commit message of the relevant change. Including as much information as possible in your bug report (including a unified diff or a merge request of suggested file changes) will ensure the issue can be progressed as rapidly as possible. Issues must be labeled "Waiver" (TODO!) and identify the version of the CTS and affected tests. Conformance Criteria -------------------- Conformance run is considered passing if all tests finish with allowed result codes. The test results are contained in the TestResults.qpa log. Each test case section contains XML tag Result, for example: Not validated The result code is the value of the StatusCode attribute. Following status codes are allowed: Pass, NotSupported, QualityWarning, CompatibilityWarning TODO: Create script for verifying logs. Vulkan platform port -------------------- Vulkan support from Platform implementation requires providing getVulkanPlatform() method in tcu::Platform class implementation. See framework/common/tcuPlatform.hpp and examples in framework/platform/win32/tcuWin32Platform.cpp and framework/platform/android/tcuAndroidPlatform.cpp. Null (dummy) driver ------------------- For testing and development purposes it might be useful to be able to run tests on dummy Vulkan implementation. One such implementation is provided in vkNullDriver.cpp. To use that, implement vk::Platform::createLibrary() with vk::createNullDriver(). Cherry GUI ---------- Vulkan test module can be used with Cherry (GUI for test execution and analysis). Cherry is available at https://android.googlesource.com/platform/external/cherry. Please follow instructions in README to get started. To enable support for Vulkan tests, dEQP-VK module must be added to list of test packages. In cherry/testrunner.go, add following line to testPackageDescriptors list (line 608 in NewTestRunner function): {"dEQP-VK", "deqp-vk", "../external/vulkancts/modules/vulkan", dataDir + "dEQP-VK-cases.xml"}, Before first launch, and every time test hierarchy has been modified, test case list must be refreshed by running: $ python scripts/build_caselists.py path/to/cherry/data Cherry must be restarted for the case list update to take effect.