4 This document describes how to build and run Vulkan Conformance Test suite.
6 Vulkan CTS is built on dEQP framework. dEQP documentation is available
7 at http://source.android.com/devices/graphics/testing.html
15 The following tools must be installed and present in the PATH variable:
17 * Git (for checking out sources)
18 * Python 3.x (for the build related scripts, some other scripts still use Python 2.7.x)
19 * CMake 2.8 (3.6 for Android NDK r17+ builds) or newer
23 * Visual Studio 2015 or newer (glslang uses several C++11 features)
27 * Standard toolchain (make, gcc/clang)
31 * Android NDK r15c or later.
32 * Android SDK with: SDK Tools, SDK Platform-tools, SDK Build-tools, and API 28
33 * Java Development Kit (JDK)
34 * Windows: either NMake or Ninja in PATH
36 If you have downloaded Android SDK tools, you can install necessary components
39 tools/android update sdk --no-ui --all --filter tools,platform-tools,build-tools-25.0.2,android-28
45 To build dEQP, you need first to download sources for zlib, libpng, glslang,
46 spirv-headers, and spirv-tools.
48 To download sources, run:
50 python external/fetch_sources.py
52 You may need to re-run `fetch_sources.py` to update to the latest glslang and
53 spirv-tools revisions occasionally.
55 With CMake out-of-source builds are always recommended. Create a build directory
56 of your choosing, and in that directory generate Makefiles or IDE project
62 cmake <path to vulkancts> -G"Visual Studio 14"
63 start dEQP-Core-default.sln
67 cmake <path to vulkancts> -G"Visual Studio 14 Win64"
68 start dEQP-Core-default.sln
70 ### Linux 32-bit Debug
72 cmake <path to vulkancts> -DCMAKE_BUILD_TYPE=Debug -DCMAKE_C_FLAGS=-m32 -DCMAKE_CXX_FLAGS=-m32
75 Release build can be done by using -DCMAKE_BUILD_TYPE=Release
77 ### Linux 64-bit Debug
79 cmake <path to vulkancts> -DCMAKE_BUILD_TYPE=Debug -DCMAKE_C_FLAGS=-m64 -DCMAKE_CXX_FLAGS=-m64
84 Following command will build dEQP.apk:
86 python scripts/android/build_apk.py --sdk <path to Android SDK> --ndk <path to Android NDK>
88 By default the CTS package will be built for the Android API level 28.
89 Another API level may be supplied using --native-api command line option.
91 The package can be installed by either running:
93 python scripts/android/install_apk.py
95 By default the CTS package will contain libdeqp.so built for armeabi-v7a, arm64-v8a,
96 x86, and x86_64 ABIs, but that can be changed using --abis command line option.
98 To pick which ABI to use at install time, following commands must be used
101 adb install --abi <ABI name> <build-root>/package/dEQP.apk /data/local/tmp/dEQP-debug.apk
107 Current mustpass is checked into repository and can be found at:
109 external/vulkancts/mustpass/1.1.4/vk-default.txt
111 Vulkan CTS mustpass can be re-generated by running:
113 python <vulkancts>/external/vulkancts/scripts/build_mustpass.py
116 Pre-compiling SPIR-V binaries
117 -----------------------------
119 For distribution, and platforms that don't support GLSL to SPIR-V compilation,
120 SPIR-V binaries can be pre-built with following command:
122 python external/vulkancts/scripts/build_spirv_binaries.py
124 By default the script builds SPIR-V binaries for Vulkan 1.1.
125 Binaries for other Vulkan versions can be requested by supplying
126 an extra command line option:
128 python external/vulkancts/scripts/build_spirv_binaries.py --target-vulkan-version <Vulkan version>
130 Binaries will be written to `external/vulkancts/data/vulkan/prebuilt/`.
132 Test modules (or in case of Android, the APK) must be re-built after building
133 SPIR-V programs in order for the binaries to be available.
139 Following command line options MUST be used when running CTS:
141 --deqp-caselist-file=<vulkancts>/external/vulkancts/mustpass/1.1.4/vk-default.txt
142 --deqp-log-images=disable
143 --deqp-log-shader-sources=disable
145 In addition on multi-device systems the device for which conformance is claimed
146 can be selected with:
148 --deqp-vk-device-id=<value>
150 To speed up the conformance run on some platforms the following command line
151 option may be used to disable frequent fflush() calls to the output logs:
153 --deqp-log-flush=disable
155 By default, the test log will be written into the path "TestResults.qpa". If the
156 platform requires a different path, it can be specified with:
158 --deqp-log-filename=<path>
160 By default, the shader cache will be written into the path "shadercache.bin". If the
161 platform requires a different path, it can be specified with:
163 --deqp-shadercache-filename=<path>
165 If the shader cache is not desired, it can be disabled with:
167 --deqp-shadercache=disable
169 No other command line options are allowed.
173 cd <builddir>/external/vulkancts/modules/vulkan
174 Debug/deqp-vk.exe --deqp-caselist-file=...
176 Test log will be written into TestResults.qpa
180 cd <builddir>/external/vulkancts/modules/vulkan
181 ./deqp-vk --deqp-vk-caselist-file=...
185 adb push <vulkancts>/external/vulkancts/mustpass/1.1.4/vk-default.txt /sdcard/vk-default.txt
190 am start -n com.drawelements.deqp/android.app.NativeActivity -e cmdLine "deqp --deqp-caselist-file=/sdcard/vk-default.txt --deqp-log-images=disable --deqp-log-shader-sources=disable --deqp-log-filename=/sdcard/TestResults.qpa"
192 Test progress will be written to device log and can be displayed with:
196 Test log will be written into `/sdcard/TestResults.qpa`.
199 Conformance Submission Package Requirements
200 -------------------------------------------
202 The conformance submission package must contain the following:
204 1. Full test logs (`TestResults.qpa`) from CTS runs against all driver builds
205 2. Result of `git status` and `git log` from CTS source directory
206 3. Any patches used on top of release tag
207 4. Conformance statement
209 Test logs (1) should be named `<submission pkg dir>/TestResults-<driver build type>.qpa`,
210 for example `TestResults-armeabi-v7a.qpa`. On platforms where multiple different driver
211 builds (for example 64-bit and 32-bit) are present, CTS logs must be provided
212 for each driver build as part of the submission package.
214 Test logs generated on a system which exposes more than one physical device
215 in a device group can be used for products that expose one or more physical
216 devices in their device group.
218 The CTS build must always be done from clean git repository that doesn't have any
219 uncommitted changes. Thus it is necessary to run and capture output of `git
220 status` and `git log` (2) in the source directory:
222 git status > <submission pkg dir>/git-status.txt
223 git log --first-parent <release tag>^..HEAD > <submission pkg dir>/git-log.txt
225 Any changes made to CTS must be committed to the local repository, and provided
226 as part of the submission package (3). This can be done by running:
228 git format-patch -o <submission pkg dir> <release tag>..HEAD
230 In general, bugfixes and changes to platform-specific code (mostly under
231 `framework/platform`) are allowed. The commit message for each change must
232 include a clear description of the change and why it is necessary. Non-porting
233 related changes must be accompanied by a waiver (see below).
235 NOTE: When cherry-picking patches on top of release tag, please use `git cherry-pick -x`
236 to include original commit hash in the commit message.
238 Conformance statement (4) must be included in a file called `STATEMENT-<adopter>`
239 and must contain following:
241 CONFORM_VERSION: <git tag of CTS release>
242 PRODUCT: <string-value>
246 Note that product/cpu/os information is also captured in `dEQP-VK.info.*` tests
247 if `vk::Platform::describePlatform()` is implemented.
249 If the submission package covers multiple products, you can list them by appending
250 additional `PRODUCT:` lines to the conformance statement. For example:
252 CONFORM_VERSION: vulkan-cts-1.1.4.0
257 The actual submission package consists of the above set of files which must
258 be bundled into a gzipped tar file named `VK<API major><API minor>_<adopter><_info>.tgz`.
259 `<API major>` is the major version of the Vulkan API specification, `<API minor>`is the minor
260 version of the Vulkan API specification.
261 `<adopter>` is the name of the Adopting member company, or some recognizable abbreviation.
262 The `<_info>` field is optional. It may be used to uniquely identify a submission
263 by OS, platform, date, or other criteria when making multiple submissions.
264 For example, a company XYZ may make a submission for a Vulkan 1.1 implementation named
265 `VK11_XYZ_PRODUCTA_Windows10.tgz`
267 One way to create a suiteable gzipped tar file is to execute the command:
269 tar -cvzf <filename.tgz> -C <submission pkg dir> .
271 where `<submission pkg dir>` is the directory containing the files from (1)-(4)
272 from above. A submission package must contain all of the files listed above,
273 and only those files.
275 As an example submission package could contain:
280 0001-Remove-Waived-Filtering-Tests.patch
281 0002-Fix-Pipeline-Parameters.patch
282 TestResults-armeabi-v7a.qpa
283 TestResults-arm64-v8a.qpa
289 The process for requesting a waiver is to report the issue by filing a bug
290 report in the Gitlab VulkanCTS project (TODO Github?). When creating the
291 submission package, include references to the waiver in the commit message of
292 the relevant change. Including as much information as possible in your bug
293 report (including a unified diff or a merge request of suggested file changes)
294 will ensure the issue can be progressed as rapidly as possible. Issues must
295 be labeled "Waiver" (TODO!) and identify the version of the CTS and affected
301 Conformance run is considered passing if all tests finish with allowed result
302 codes. Test results are contained in the TestResults.qpa log. Each
303 test case section contains XML tag Result, for example:
305 <Result StatusCode="Pass">Not validated</Result>
307 The result code is the value of the StatusCode attribute. Following status
315 Submission package can be verified using `external/vulkancts/scripts/verify_submission.py`
316 script. The script takes two arguments: path to extracted submission package
317 and path to current mustpass list. For example:
319 python external/vulkancts/scripts/verify_submission.py VK_11_Khronos_1/ external/vulkancts/mustpass/1.1.4/vk-default.txt
321 Please note that the script reports a warning even for a correctly generated git-log.txt
322 If your git-log.txt contains only head commit of the release tag then
323 the warning can be ignored.
328 Vulkan support from Platform implementation requires providing
329 `getVulkanPlatform()` method in `tcu::Platform` class implementation.
331 See `framework/common/tcuPlatform.hpp` and examples in
332 `framework/platform/win32/tcuWin32Platform.cpp` and
333 `framework/platform/android/tcuAndroidPlatform.cpp`.
335 If any WSI extensions are supported, platform port must also implement
336 methods for creating native display (`vk::Platform::createWsiDisplay`)
337 and window handles (`vk::wsi::Display::createWindow`). Otherwise tests
338 under `dEQP-VK.wsi` will fail.
344 For testing and development purposes it might be useful to be able to run
345 tests on dummy Vulkan implementation. One such implementation is provided in
346 vkNullDriver.cpp. To use that, implement `vk::Platform::createLibrary()` with
347 `vk::createNullDriver()`.
353 Vulkan CTS framework includes first-party support for validation layers, that
354 can be turned on with `--deqp-validation=enable` command line option.
356 When validation is turned on, default instance and device will be created with
357 validation layers enabled and debug callback is registered to record any
358 messages. Debug messages collected during test execution will be included at
359 the end of the test case log.
361 If any validation errors are found, test result will be set to `InternalError`.
363 By default `VK_DEBUG_REPORT_INFORMATION_BIT_EXT` and `_DEBUG_BIT_EXT` messages
364 are excluded from the log, but that can be customized by modifying
365 `vkt::TestCaseExecutor::deinit()` in `vktTestPackage.cpp`.
367 On the Android target, layers can be added to the APK during the build process
368 by setting the `--layers-path` command line option to point into the NDK or to
369 a locally-built layers tree. The layers are expected to be found under $abi/
370 under the layers path.
376 Vulkan test module can be used with Cherry (GUI for test execution and
377 analysis). Cherry is available at
378 https://android.googlesource.com/platform/external/cherry. Please follow
379 instructions in README to get started.
381 Before first launch, and every time test hierarchy has been modified, test
382 case list must be refreshed by running:
384 python scripts/build_caselists.py path/to/cherry/data
386 Cherry must be restarted for the case list update to take effect.
392 Vulkan CTS can be optionally run with the shader optimizer enabled. This
393 is an experimental feature which can be used to further stress both the
394 drivers as well as the optimizer itself. The shader optimizer is disabled
397 The following command line options can be used to configure the shader
400 --deqp-optimization-recipe=<number>
402 The list of the optimization recipes can be found and customized in the
403 `optimizeCompiledBinary()` function in `vkPrograms.cpp`.
405 As of this writing, there are 8 recipes to choose from:
407 0. Disabled (default)
408 1. The example recipe from spir-v opt 1.0 whitepaper
409 2. RegisterPerformancePasses from commandline optimizer tool october 2017
410 3. RegisterSizePasses from commandline optimizer tool october 2017
411 4. RegisterLegalizationPasses from commandline optimizer tool April 2018
412 5. RegisterPerformancePasses from commandline optimizer tool April 2018
413 6. RegisterPerformancePasses from commandline optimizer tool April 2018 with CreateCommonUniformElimPass
414 7. RegisterSizePasses from commandline optimizer tool April 2018
415 8. RegisterSizePasses from commandline optimizer tool April 2018 with CreateCommonUniformElimPass
417 The `Register...Passes()` calls change in the SPIR-V optimizer tool from
418 time to time. Since different sets of passes may result in different
419 shaders, having several fixed sets is useful for issue discovery.
421 --deqp-optimize-spirv=enable
423 This option is not required to run the optimizer. By default, the shader
424 optimizer only optimizes shaders generated from GLSL or HLSL, and leaves
425 hand-written SPIR-V shaders alone.
427 Many of the hand-written SPIR-V tests stress specific features of the
428 SPIR-V which might get optimized out. Using this option will enable the
429 optimizer on the hand-written SPIR-V as well, which may be useful in
430 finding new bugs in drivers or the optimizer itself, but will likely
431 invalidate the tests themselves.
437 The Vulkan CTS framework contains a shader cache for speeding up the running
438 of the CTS. Skipping shader compilation can significantly reduce runtime,
439 especially for repeated runs.
441 Default behavior is to have the shader cache enabled, but truncated at the
442 start of the CTS run. This still gives the benefit of skipping shader
443 compilation for identical shaders in different tests (which there are many),
444 while making sure that the shader cache file does not grow indefinitely.
446 The shader cache identifies the shaders by hashing the shader source code
447 along with various bits of information that may affect the shader compilation
448 (such as shader stage, CTS version, possible compilation flags, etc). If a
449 cached shader with matching hash is found, a byte-by-byte comparison of the
450 shader sources is made to make sure that the correct shader is being
451 retrieved from the cache.
453 The behavior of the shader cache can be modified with the following command
456 --deqp-shadercache=disable
458 Disable the shader cache. All shaders will be compiled every time.
460 --deqp-shadercache-filename=<filename>
462 Set the name of the file where the cached shaders will be stored. This
463 option may be required for the shader cache to work at all on Android
466 --deqp-shadercache-truncate=disable
468 Do not truncate the shader cache file at startup. No shader compilation will
469 occur on repeated runs of the CTS.
474 The RenderDoc (https://renderdoc.org/) graphics debugger may be used to debug
477 Following command line option should be used when launching tests from
480 --deqp-renderdoc=enable
482 This causes the framework to interface with the debugger and mark each dEQP
483 test case as a separate 'frame', just for the purpose of capturing. The frames
484 are added using RenderDoc 'In-Application API', instead of swapchain operations.