Vulkan CTS README ================= This document describes how to build and run Vulkan Conformance Test suite. Vulkan CTS is built on dEQP framework. dEQP documentation is available at http://source.android.com/devices/graphics/testing.html Requirements ------------ ### Common The following tools must be installed and present in the PATH variable: * Git (for checking out sources) * Python 3.x (for the build related scripts, some other scripts still use Python 2.7.x) * CMake 3.10.2 or newer ### Win32 * Visual Studio 2015 or newer (glslang uses several C++11 features) ### Linux * Standard toolchain (make, gcc/clang) * If you have X11 installed, then the build assumes you also have the `GL/glx.h` header file. * You can get this from the `mesa-common-dev` Ubuntu package. ### MacOS * Xcode Vulkan SDK for macOS includes a MoltenVK runtime library which is recommended method for developing a Vulkan application. ### Android * Android NDK r17c or later. * Android SDK with: SDK Tools, SDK Platform-tools, SDK Build-tools, and API 28 * Java Development Kit (JDK) * Windows: either NMake or Ninja in PATH If you have downloaded the Android SDK command line tools package (25.2.3 or higher) then you can install the necessary components by running: tools/bin/sdkmanager tools platform-tools 'build-tools;25.0.2' 'platforms;android-28' Building CTS ------------ To build dEQP, you need first to download sources for zlib, libpng, glslang, vulkan-docs, spirv-headers, and spirv-tools. To download sources, run: python external/fetch_sources.py You may need to re-run `fetch_sources.py` to update to the latest glslang, vulkan-docs and spirv-tools revisions occasionally. You also need to install lxml python module by running: python -m pip install lxml With CMake out-of-source builds are always recommended. Create a build directory of your choosing, and in that directory generate Makefiles or IDE project using cmake. ### Windows x86-32 cmake -G"Visual Studio 14" start dEQP-Core-default.sln ### Windows x86-64 cmake -G"Visual Studio 14 Win64" start dEQP-Core-default.sln ### Linux 32-bit Debug cmake -DCMAKE_BUILD_TYPE=Debug -DCMAKE_C_FLAGS=-m32 -DCMAKE_CXX_FLAGS=-m32 make -j Release build can be done by using -DCMAKE_BUILD_TYPE=Release If building for 32-bit x86 with GCC, you probably also want to add `-msse2 -mfpmath=sse` to ensure that you get correct IEEE floating-point behavior. ### Linux 64-bit Debug cmake -DCMAKE_BUILD_TYPE=Debug -DCMAKE_C_FLAGS=-m64 -DCMAKE_CXX_FLAGS=-m64 make -j ### MacOS 64-bit Debug cmake -DCMAKE_BUILD_TYPE=Debug -DDEQP_TARGET=osx -DCMAKE_C_FLAGS=-m64 -DCMAKE_CXX_FLAGS=-m64 make -j ### Android Following command will build dEQP.apk: python scripts/android/build_apk.py --sdk --ndk By default the CTS package will be built for the Android API level 28. Another API level may be supplied using --native-api command line option. The package can be installed by either running: python scripts/android/install_apk.py By default the CTS package will contain libdeqp.so built for armeabi-v7a, arm64-v8a, x86, and x86_64 ABIs, but that can be changed at build time by passing the --abis command line option to `scripts/android/build_apk.py`. To pick which ABI to use at _install time_, use the following command instead: adb install -g --abi /package/dEQP.apk Building Mustpass ----------------- Current mustpass is checked into repository and can be found at: external/vulkancts/mustpass/master/vk-default.txt This file contains list of files located in vk-default subdirectory. Those files contain tests from top-level groups and together they contain all test cases that should pass. Vulkan CTS mustpass can be re-generated by running: python /external/vulkancts/scripts/build_mustpass.py Running CTS ----------- Following command line options MUST be used when running CTS: --deqp-caselist-file=/external/vulkancts/mustpass/master/vk-default.txt --deqp-log-images=disable --deqp-log-shader-sources=disable In addition on multi-device systems the device for which conformance is claimed can be selected with: --deqp-vk-device-id= To speed up the conformance run on some platforms the following command line option may be used to disable frequent fflush() calls to the output logs: --deqp-log-flush=disable By default, the test log will be written into the path "TestResults.qpa". If the platform requires a different path, it can be specified with: --deqp-log-filename= By default, the CTS will expect to find its test resource files in the current working directory. This can be overridden with: --deqp-archive-dir= By default, the shader cache will be written into the path "shadercache.bin". If the platform requires a different path, it can be specified with: --deqp-shadercache-filename= If the shader cache is not desired, it can be disabled with: --deqp-shadercache=disable CTS execution may be split into N fractions ( for the purpose of running it in parallel ) using --deqp-fraction=I,N where I denotes index of current CTS execution ( I=[0..N-1], N=[1..8] ) When collecting results for a Conformance Submission Package the number of fractions must not exceed 8, and a list of mandatory information tests for each fraction must be supplied: --deqp-fraction-mandatory-caselist-file=external/vulkancts/mustpass/master/vk-fraction-mandatory-tests.txt To specify file containing waived tests that are omitted only by specified vendor and renderer/device the following command line option may be used: --deqp-waiver-file= To make log more clear for high level shader source analysis output of decompiled SPIR-V shader sources can be disabled: --deqp-log-decompiled-spirv=disable Also non-informative empty LogInfo sections can be removed from output into log by specifying: --deqp-log-empty-loginfo=disable No other command line options are allowed. ### Win32 cd /external/vulkancts/modules/vulkan Debug\deqp-vk.exe --deqp-caselist-file=... Test log will be written into TestResults.qpa ### Linux cd /external/vulkancts/modules/vulkan ./deqp-vk --deqp-caselist-file=... ### MacOS cd /external/vulkancts/modules/vulkan ./deqp-vk --deqp-caselist-file=... ### Android adb push /external/vulkancts/mustpass/master/vk-default.txt /sdcard/vk-default.txt adb shell In device shell: am start -n com.drawelements.deqp/android.app.NativeActivity -e cmdLine "deqp --deqp-caselist-file=/sdcard/vk-default.txt --deqp-log-images=disable --deqp-log-shader-sources=disable --deqp-log-filename=/sdcard/TestResults.qpa" Test progress will be written to device log and can be displayed with: adb logcat -s dEQP Test log will be written into `/sdcard/TestResults.qpa`. Conformance Submission Package Requirements ------------------------------------------- The conformance submission package must contain the following: 1. Full test logs (`TestResults.qpa`) from CTS runs against all driver builds and all fractions 2. Result of `git status` and `git log` from CTS source directory 3. Any patches used on top of release tag 4. Conformance statement Test logs (1) should be named `/TestResults---of-.qpa`, for example `TestResults-armeabi-v7a-1-of-8.qpa`. On platforms where multiple different driver builds (for example 64-bit and 32-bit) are present, CTS logs must be provided for each driver build as part of the submission package. If CTS run was split into multiple fractions then result files for all fractions must be provided, each file must contain results of the mandatory information tests. Fractions may be run on different physical devices but each device must represent the same Conformant Product. Test logs generated on a system which exposes more than one physical device in a device group can be used for products that expose one or more physical devices in their device group. The CTS build must always be done from clean git repository that doesn't have any uncommitted changes. Thus it is necessary to run and capture output of `git status` and `git log` (2) in the source directory: git status > /git-status.txt git log --first-parent ^..HEAD > /git-log.txt Any changes made to CTS must be committed to the local repository, and provided as part of the submission package (3). This can be done by running: git format-patch -o ..HEAD Changes to platform-specific code (mostly under `framework/platform`) are allowed. The commit message for each change must include a clear description of the change and why it is necessary. Bugfixes to the tests are allowed. Before being used for a submission, bugfixes must be accepted and merged into the CTS repository. `git cherry-pick` is strongly recommended as a method of applying bug fixes. Other changes must be accompanied by a waiver (see below). NOTE: When cherry-picking patches on top of release tag, please use `git cherry-pick -x` to include original commit hash in the commit message. Conformance statement (4) must be included in a file called `STATEMENT-` and must contain following: CONFORM_VERSION: PRODUCT: CPU: OS: Note that product/cpu/os information is also captured in `dEQP-VK.info.*` tests if `vk::Platform::describePlatform()` is implemented. If the submission package covers multiple products, you can list them by appending additional `PRODUCT:` lines to the conformance statement. For example: CONFORM_VERSION: vulkan-cts-1.2.6.0 PRODUCT: Product A PRODUCT: Product B ... The actual submission package consists of the above set of files which must be bundled into a gzipped tar file named `VK_<_info>.tgz`. `` is the major version of the Vulkan API specification, ``is the minor version of the Vulkan API specification. `` is the name of the Adopting member company, or some recognizable abbreviation. The `<_info>` field is optional. It may be used to uniquely identify a submission by OS, platform, date, or other criteria when making multiple submissions. For example, a company XYZ may make a submission for a Vulkan 1.1 implementation named `VK11_XYZ_PRODUCTA_Windows10.tgz` One way to create a suiteable gzipped tar file is to execute the command: tar -cvzf -C . where `` is the directory containing the files from (1)-(4) from above. A submission package must contain all of the files listed above, and only those files. As an example submission package could contain: STATEMENT-Khronos git-log.txt git-status.txt 0001-Remove-Waived-Filtering-Tests.patch 0002-Fix-Pipeline-Parameters.patch TestResults-armeabi-v7a.qpa TestResults-arm64-v8a.qpa Waivers ------- The process for requesting a waiver is to report the issue by filing a bug report in the Gitlab VulkanCTS project (TODO Github?). When creating the submission package, include references to the waiver in the commit message of the relevant change. Including as much information as possible in your bug report (including a unified diff or a merge request of suggested file changes) will ensure the issue can be progressed as rapidly as possible. Issues must be labeled "Waiver" (TODO!) and identify the version of the CTS and affected tests. Conformance Criteria -------------------- Conformance run is considered passing if all tests finish with allowed result codes. Test results are contained in the TestResults.qpa log. Each test case section contains XML tag Result, for example: Not validated The result code is the value of the StatusCode attribute. Following status codes are allowed: Pass NotSupported QualityWarning CompatibilityWarning Waiver Submission package can be verified using `verify_submission.py` script located in [VK-GL-CTS-Tools](https://github.com/KhronosGroup/VK-GL-CTS-Tools). Vulkan platform port -------------------- Vulkan support from Platform implementation requires providing `getVulkanPlatform()` method in `tcu::Platform` class implementation. See `framework/common/tcuPlatform.hpp` and examples in `framework/platform/win32/tcuWin32Platform.cpp` and `framework/platform/android/tcuAndroidPlatform.cpp`. If any WSI extensions are supported, platform port must also implement methods for creating native display (`vk::Platform::createWsiDisplay`) and window handles (`vk::wsi::Display::createWindow`). Otherwise tests under `dEQP-VK.wsi` will fail. Null (dummy) driver ------------------- For testing and development purposes it might be useful to be able to run tests on dummy Vulkan implementation. One such implementation is provided in vkNullDriver.cpp. To use that, implement `vk::Platform::createLibrary()` with `vk::createNullDriver()`. Validation Layers ----------------- Vulkan CTS framework includes first-party support for validation layers, that can be turned on with `--deqp-validation=enable` command line option. When validation is turned on, default instance and device will be created with validation layers enabled and debug callback is registered to record any messages. Debug messages collected during test execution will be included at the end of the test case log. In addition, when the `--deqp-print-validation-errors` command line option is used, validation errors are additionally printed to standard error in the moment they are generated. If any validation errors are found, test result will be set to `InternalError`. By default `VK_DEBUG_REPORT_INFORMATION_BIT_EXT` and `_DEBUG_BIT_EXT` messages are excluded from the log, but that can be customized by modifying `vk::DebugReportMessage::shouldBeLogged()` in `vkDebugReportUtil.hpp`. On the Android target, layers can be added to the APK during the build process by setting the `--layers-path` command line option to point into the NDK or to a locally-built layers tree. The layers are expected to be found under $abi/ under the layers path. Cherry GUI ---------- Vulkan test module can be used with Cherry (GUI for test execution and analysis). Cherry is available at https://android.googlesource.com/platform/external/cherry. Please follow instructions in README to get started. Before first launch, and every time test hierarchy has been modified, test case list must be refreshed by running: python scripts/build_caselists.py path/to/cherry/data Cherry must be restarted for the case list update to take effect. Shader Optimizer ---------------- Vulkan CTS can be optionally run with the shader optimizer enabled. This is an experimental feature which can be used to further stress both the drivers as well as the optimizer itself. The shader optimizer is disabled by default. The following command line options can be used to configure the shader optimizer: --deqp-optimization-recipe= The list of the optimization recipes can be found and customized in the `optimizeCompiledBinary()` function in `vkPrograms.cpp`. As of this writing, there are 2 recipes to choose from: 0. Disabled (default) 1. Optimize for performance 2. Optimize for size The performance and size optimization recipes are defined by the spir-v optimizer, and will change from time to time as the optimizer matures. --deqp-optimize-spirv=enable This option is not required to run the optimizer. By default, the shader optimizer only optimizes shaders generated from GLSL or HLSL, and leaves hand-written SPIR-V shaders alone. Many of the hand-written SPIR-V tests stress specific features of the SPIR-V which might get optimized out. Using this option will enable the optimizer on the hand-written SPIR-V as well, which may be useful in finding new bugs in drivers or the optimizer itself, but will likely invalidate the tests themselves. Shader Cache ------------ The Vulkan CTS framework contains a shader cache for speeding up the running of the CTS. Skipping shader compilation can significantly reduce runtime, especially for repeated runs. Default behavior is to have the shader cache enabled, but truncated at the start of the CTS run. This still gives the benefit of skipping shader compilation for identical shaders in different tests (which there are many), while making sure that the shader cache file does not grow indefinitely. The shader cache identifies the shaders by hashing the shader source code along with various bits of information that may affect the shader compilation (such as shader stage, CTS version, possible compilation flags, etc). If a cached shader with matching hash is found, a byte-by-byte comparison of the shader sources is made to make sure that the correct shader is being retrieved from the cache. The behavior of the shader cache can be modified with the following command line options: --deqp-shadercache=disable Disable the shader cache. All shaders will be compiled every time. --deqp-shadercache-filename= Set the name of the file where the cached shaders will be stored. This option may be required for the shader cache to work at all on Android targets. --deqp-shadercache-truncate=disable Do not truncate the shader cache file at startup. No shader compilation will occur on repeated runs of the CTS. RenderDoc --------- The RenderDoc (https://renderdoc.org/) graphics debugger may be used to debug Vulkan tests. Following command line option should be used when launching tests from RenderDoc UI: --deqp-renderdoc=enable This causes the framework to interface with the debugger and mark each dEQP test case as a separate 'frame', just for the purpose of capturing. The frames are added using RenderDoc 'In-Application API', instead of swapchain operations. Third Party Runners ------------------- Some CTS tests use third-party runners. By default all tests are executed regardless of runner type (`any`). To exclude all tests using any of the external runners (`none`) or to only include tests using a certain runner: --deqp-runner-type=(any|none|amber) Command Line ------------ Full list of parameters for the `deqp-vk` module: OpenGL and OpenCL parameters not affecting Vulkan API were suppressed. -h, --help Show this help -n, --deqp-case= Test case(s) to run, supports wildcards (e.g. dEQP-GLES2.info.*) --deqp-caselist= Case list to run in trie format (e.g. {dEQP-GLES2{info{version,renderer}}}) --deqp-caselist-file= Read case list (in trie format) from given file --deqp-caselist-resource= Read case list (in trie format) from given file located application's assets --deqp-stdin-caselist Read case list (in trie format) from stdin --deqp-log-filename= Write test results to given file default: 'TestResults.qpa' --deqp-runmode=[execute|xml-caselist|txt-caselist|stdout-caselist] Execute tests, or write list of test cases into a file default: 'execute' --deqp-caselist-export-file= Set the target file name pattern for caselist export default: '${packageName}-cases.${typeExtension}' --deqp-watchdog=[enable|disable] Enable test watchdog default: 'disable' --deqp-crashhandler=[enable|disable] Enable crash handling default: 'disable' --deqp-base-seed= Base seed for test cases that use randomization default: '0' --deqp-test-iteration-count= Iteration count for cases that support variable number of iterations default: '0' --deqp-visibility=[windowed|fullscreen|hidden] Default test window visibility default: 'windowed' --deqp-surface-width= Use given surface width if possible default: '-1' --deqp-surface-height= Use given surface height if possible default: '-1' --deqp-surface-type=[window|pixmap|pbuffer|fbo] Use given surface type default: 'window' --deqp-screen-rotation=[unspecified|0|90|180|270] Screen rotation for platforms that support it default: '0' --deqp-vk-device-id= Vulkan device ID (IDs start from 1) default: '1' --deqp-vk-device-group-id= Vulkan device Group ID (IDs start from 1) default: '1' --deqp-log-images=[enable|disable] Enable or disable logging of result images default: 'enable' --deqp-log-shader-sources=[enable|disable] Enable or disable logging of shader sources default: 'enable' --deqp-test-oom=[enable|disable] Run tests that exhaust memory on purpose default: 'enable' --deqp-archive-dir= Path to test resource files default: '.' --deqp-log-flush=[enable|disable] Enable or disable log file fflush default: 'enable' --deqp-validation=[enable|disable] Enable or disable test case validation default: 'disable' --deqp-print-validation-errors Print validation errors to standard error --deqp-optimization-recipe= Shader optimization recipe (0=disabled, 1=performance, 2=size) default: '0' --deqp-optimize-spirv=[enable|disable] Apply optimization to spir-v shaders as well default: 'disable' --deqp-shadercache=[enable|disable] Enable or disable shader cache default: 'enable' --deqp-shadercache-filename= Write shader cache to given file default: 'shadercache.bin' --deqp-shadercache-truncate=[enable|disable] Truncate shader cache before running tests default: 'enable' --deqp-renderdoc=[enable|disable] Enable RenderDoc frame markers default: 'disable' --deqp-fraction= Run a fraction of the test cases (e.g. N,M means run group%M==N) default: '' --deqp-fraction-mandatory-caselist-file= Case list file that must be run for each fraction default: '' --deqp-waiver-file= Read waived tests from given file default: '' --deqp-runner-type=[any|none|amber] Filter test cases based on runner default: 'any' --deqp-terminate-on-fail=[enable|disable] Terminate the run on first failure default: 'disable'