4 This document describes how to build and run Vulkan Conformance Test suite.
6 Vulkan CTS is built on dEQP framework. dEQP documentation is available
7 at http://source.android.com/devices/graphics/testing.html
15 The following tools must be installed and present in the PATH variable:
17 * Git (for checking out sources)
18 * Python 3.x (for the build related scripts, some other scripts still use Python 2.7.x)
19 * CMake 3.0 (3.6 for Android NDK r17+ builds) or newer
23 * Visual Studio 2015 or newer (glslang uses several C++11 features)
27 * Standard toolchain (make, gcc/clang)
31 * Android NDK r15c or later.
32 * Android SDK with: SDK Tools, SDK Platform-tools, SDK Build-tools, and API 28
33 * Java Development Kit (JDK)
34 * Windows: either NMake or Ninja in PATH
36 If you have downloaded Android SDK tools, you can install necessary components
39 tools/android update sdk --no-ui --all --filter tools,platform-tools,build-tools-25.0.2,android-28
45 To build dEQP, you need first to download sources for zlib, libpng, glslang,
46 spirv-headers, and spirv-tools.
48 To download sources, run:
50 python external/fetch_sources.py
52 You may need to re-run `fetch_sources.py` to update to the latest glslang and
53 spirv-tools revisions occasionally.
55 With CMake out-of-source builds are always recommended. Create a build directory
56 of your choosing, and in that directory generate Makefiles or IDE project
62 cmake <path to vulkancts> -G"Visual Studio 14"
63 start dEQP-Core-default.sln
67 cmake <path to vulkancts> -G"Visual Studio 14 Win64"
68 start dEQP-Core-default.sln
70 ### Linux 32-bit Debug
72 cmake <path to vulkancts> -DCMAKE_BUILD_TYPE=Debug -DCMAKE_C_FLAGS=-m32 -DCMAKE_CXX_FLAGS=-m32
75 Release build can be done by using -DCMAKE_BUILD_TYPE=Release
77 If building for 32-bit x86 with GCC, you probably also want to add `-msse2
78 -mfpmath=sse` to ensure that you get correct IEEE floating-point behavior.
80 ### Linux 64-bit Debug
82 cmake <path to vulkancts> -DCMAKE_BUILD_TYPE=Debug -DCMAKE_C_FLAGS=-m64 -DCMAKE_CXX_FLAGS=-m64
87 Following command will build dEQP.apk:
89 python scripts/android/build_apk.py --sdk <path to Android SDK> --ndk <path to Android NDK>
91 By default the CTS package will be built for the Android API level 28.
92 Another API level may be supplied using --native-api command line option.
94 The package can be installed by either running:
96 python scripts/android/install_apk.py
98 By default the CTS package will contain libdeqp.so built for armeabi-v7a, arm64-v8a,
99 x86, and x86_64 ABIs, but that can be changed using --abis command line option.
101 To pick which ABI to use at install time, following commands must be used
104 adb install --abi <ABI name> <build-root>/package/dEQP.apk /data/local/tmp/dEQP-debug.apk
110 Current mustpass is checked into repository and can be found at:
112 external/vulkancts/mustpass/master/vk-default.txt
114 Vulkan CTS mustpass can be re-generated by running:
116 python <vulkancts>/external/vulkancts/scripts/build_mustpass.py
122 Following command line options MUST be used when running CTS:
124 --deqp-caselist-file=<vulkancts>/external/vulkancts/mustpass/master/vk-default.txt
125 --deqp-log-images=disable
126 --deqp-log-shader-sources=disable
128 In addition on multi-device systems the device for which conformance is claimed
129 can be selected with:
131 --deqp-vk-device-id=<value>
133 To speed up the conformance run on some platforms the following command line
134 option may be used to disable frequent fflush() calls to the output logs:
136 --deqp-log-flush=disable
138 By default, the test log will be written into the path "TestResults.qpa". If the
139 platform requires a different path, it can be specified with:
141 --deqp-log-filename=<path>
143 By default, the shader cache will be written into the path "shadercache.bin". If the
144 platform requires a different path, it can be specified with:
146 --deqp-shadercache-filename=<path>
148 If the shader cache is not desired, it can be disabled with:
150 --deqp-shadercache=disable
152 No other command line options are allowed.
156 cd <builddir>/external/vulkancts/modules/vulkan
157 Debug\deqp-vk.exe --deqp-caselist-file=...
159 Test log will be written into TestResults.qpa
163 cd <builddir>/external/vulkancts/modules/vulkan
164 ./deqp-vk --deqp-vk-caselist-file=...
168 adb push <vulkancts>/external/vulkancts/mustpass/master/vk-default.txt /sdcard/vk-default.txt
173 am start -n com.drawelements.deqp/android.app.NativeActivity -e cmdLine "deqp --deqp-caselist-file=/sdcard/vk-default.txt --deqp-log-images=disable --deqp-log-shader-sources=disable --deqp-log-filename=/sdcard/TestResults.qpa"
175 Test progress will be written to device log and can be displayed with:
179 Test log will be written into `/sdcard/TestResults.qpa`.
182 Conformance Submission Package Requirements
183 -------------------------------------------
185 The conformance submission package must contain the following:
187 1. Full test logs (`TestResults.qpa`) from CTS runs against all driver builds
188 2. Result of `git status` and `git log` from CTS source directory
189 3. Any patches used on top of release tag
190 4. Conformance statement
192 Test logs (1) should be named `<submission pkg dir>/TestResults-<driver build type>.qpa`,
193 for example `TestResults-armeabi-v7a.qpa`. On platforms where multiple different driver
194 builds (for example 64-bit and 32-bit) are present, CTS logs must be provided
195 for each driver build as part of the submission package.
197 Test logs generated on a system which exposes more than one physical device
198 in a device group can be used for products that expose one or more physical
199 devices in their device group.
201 The CTS build must always be done from clean git repository that doesn't have any
202 uncommitted changes. Thus it is necessary to run and capture output of `git
203 status` and `git log` (2) in the source directory:
205 git status > <submission pkg dir>/git-status.txt
206 git log --first-parent <release tag>^..HEAD > <submission pkg dir>/git-log.txt
208 Any changes made to CTS must be committed to the local repository, and provided
209 as part of the submission package (3). This can be done by running:
211 git format-patch -o <submission pkg dir> <release tag>..HEAD
213 In general, bugfixes and changes to platform-specific code (mostly under
214 `framework/platform`) are allowed. The commit message for each change must
215 include a clear description of the change and why it is necessary. Non-porting
216 related changes must be accompanied by a waiver (see below).
218 NOTE: When cherry-picking patches on top of release tag, please use `git cherry-pick -x`
219 to include original commit hash in the commit message.
221 Conformance statement (4) must be included in a file called `STATEMENT-<adopter>`
222 and must contain following:
224 CONFORM_VERSION: <git tag of CTS release>
225 PRODUCT: <string-value>
229 Note that product/cpu/os information is also captured in `dEQP-VK.info.*` tests
230 if `vk::Platform::describePlatform()` is implemented.
232 If the submission package covers multiple products, you can list them by appending
233 additional `PRODUCT:` lines to the conformance statement. For example:
235 CONFORM_VERSION: vulkan-cts-1.1.6.0
240 The actual submission package consists of the above set of files which must
241 be bundled into a gzipped tar file named `VK<API major><API minor>_<adopter><_info>.tgz`.
242 `<API major>` is the major version of the Vulkan API specification, `<API minor>`is the minor
243 version of the Vulkan API specification.
244 `<adopter>` is the name of the Adopting member company, or some recognizable abbreviation.
245 The `<_info>` field is optional. It may be used to uniquely identify a submission
246 by OS, platform, date, or other criteria when making multiple submissions.
247 For example, a company XYZ may make a submission for a Vulkan 1.1 implementation named
248 `VK11_XYZ_PRODUCTA_Windows10.tgz`
250 One way to create a suiteable gzipped tar file is to execute the command:
252 tar -cvzf <filename.tgz> -C <submission pkg dir> .
254 where `<submission pkg dir>` is the directory containing the files from (1)-(4)
255 from above. A submission package must contain all of the files listed above,
256 and only those files.
258 As an example submission package could contain:
263 0001-Remove-Waived-Filtering-Tests.patch
264 0002-Fix-Pipeline-Parameters.patch
265 TestResults-armeabi-v7a.qpa
266 TestResults-arm64-v8a.qpa
272 The process for requesting a waiver is to report the issue by filing a bug
273 report in the Gitlab VulkanCTS project (TODO Github?). When creating the
274 submission package, include references to the waiver in the commit message of
275 the relevant change. Including as much information as possible in your bug
276 report (including a unified diff or a merge request of suggested file changes)
277 will ensure the issue can be progressed as rapidly as possible. Issues must
278 be labeled "Waiver" (TODO!) and identify the version of the CTS and affected
284 Conformance run is considered passing if all tests finish with allowed result
285 codes. Test results are contained in the TestResults.qpa log. Each
286 test case section contains XML tag Result, for example:
288 <Result StatusCode="Pass">Not validated</Result>
290 The result code is the value of the StatusCode attribute. Following status
298 Submission package can be verified using `external/vulkancts/scripts/verify_submission.py`
299 script. The script takes two arguments: path to extracted submission package
300 and path to current mustpass list. For example:
302 python external/vulkancts/scripts/verify_submission.py VK_11_Khronos_1/ external/vulkancts/mustpass/master/vk-default.txt
304 Please note that the script reports a warning even for a correctly generated git-log.txt
305 If your git-log.txt contains only head commit of the release tag then
306 the warning can be ignored.
311 Vulkan support from Platform implementation requires providing
312 `getVulkanPlatform()` method in `tcu::Platform` class implementation.
314 See `framework/common/tcuPlatform.hpp` and examples in
315 `framework/platform/win32/tcuWin32Platform.cpp` and
316 `framework/platform/android/tcuAndroidPlatform.cpp`.
318 If any WSI extensions are supported, platform port must also implement
319 methods for creating native display (`vk::Platform::createWsiDisplay`)
320 and window handles (`vk::wsi::Display::createWindow`). Otherwise tests
321 under `dEQP-VK.wsi` will fail.
327 For testing and development purposes it might be useful to be able to run
328 tests on dummy Vulkan implementation. One such implementation is provided in
329 vkNullDriver.cpp. To use that, implement `vk::Platform::createLibrary()` with
330 `vk::createNullDriver()`.
336 Vulkan CTS framework includes first-party support for validation layers, that
337 can be turned on with `--deqp-validation=enable` command line option.
339 When validation is turned on, default instance and device will be created with
340 validation layers enabled and debug callback is registered to record any
341 messages. Debug messages collected during test execution will be included at
342 the end of the test case log.
344 If any validation errors are found, test result will be set to `InternalError`.
346 By default `VK_DEBUG_REPORT_INFORMATION_BIT_EXT` and `_DEBUG_BIT_EXT` messages
347 are excluded from the log, but that can be customized by modifying
348 `vkt::TestCaseExecutor::deinit()` in `vktTestPackage.cpp`.
350 On the Android target, layers can be added to the APK during the build process
351 by setting the `--layers-path` command line option to point into the NDK or to
352 a locally-built layers tree. The layers are expected to be found under $abi/
353 under the layers path.
359 Vulkan test module can be used with Cherry (GUI for test execution and
360 analysis). Cherry is available at
361 https://android.googlesource.com/platform/external/cherry. Please follow
362 instructions in README to get started.
364 Before first launch, and every time test hierarchy has been modified, test
365 case list must be refreshed by running:
367 python scripts/build_caselists.py path/to/cherry/data
369 Cherry must be restarted for the case list update to take effect.
375 Vulkan CTS can be optionally run with the shader optimizer enabled. This
376 is an experimental feature which can be used to further stress both the
377 drivers as well as the optimizer itself. The shader optimizer is disabled
380 The following command line options can be used to configure the shader
383 --deqp-optimization-recipe=<number>
385 The list of the optimization recipes can be found and customized in the
386 `optimizeCompiledBinary()` function in `vkPrograms.cpp`.
388 As of this writing, there are 2 recipes to choose from:
390 0. Disabled (default)
391 1. Optimize for performance
394 The performance and size optimization recipes are defined by the spir-v
395 optimizer, and will change from time to time as the optimizer matures.
397 --deqp-optimize-spirv=enable
399 This option is not required to run the optimizer. By default, the shader
400 optimizer only optimizes shaders generated from GLSL or HLSL, and leaves
401 hand-written SPIR-V shaders alone.
403 Many of the hand-written SPIR-V tests stress specific features of the
404 SPIR-V which might get optimized out. Using this option will enable the
405 optimizer on the hand-written SPIR-V as well, which may be useful in
406 finding new bugs in drivers or the optimizer itself, but will likely
407 invalidate the tests themselves.
413 The Vulkan CTS framework contains a shader cache for speeding up the running
414 of the CTS. Skipping shader compilation can significantly reduce runtime,
415 especially for repeated runs.
417 Default behavior is to have the shader cache enabled, but truncated at the
418 start of the CTS run. This still gives the benefit of skipping shader
419 compilation for identical shaders in different tests (which there are many),
420 while making sure that the shader cache file does not grow indefinitely.
422 The shader cache identifies the shaders by hashing the shader source code
423 along with various bits of information that may affect the shader compilation
424 (such as shader stage, CTS version, possible compilation flags, etc). If a
425 cached shader with matching hash is found, a byte-by-byte comparison of the
426 shader sources is made to make sure that the correct shader is being
427 retrieved from the cache.
429 The behavior of the shader cache can be modified with the following command
432 --deqp-shadercache=disable
434 Disable the shader cache. All shaders will be compiled every time.
436 --deqp-shadercache-filename=<filename>
438 Set the name of the file where the cached shaders will be stored. This
439 option may be required for the shader cache to work at all on Android
442 --deqp-shadercache-truncate=disable
444 Do not truncate the shader cache file at startup. No shader compilation will
445 occur on repeated runs of the CTS.
450 The RenderDoc (https://renderdoc.org/) graphics debugger may be used to debug
453 Following command line option should be used when launching tests from
456 --deqp-renderdoc=enable
458 This causes the framework to interface with the debugger and mark each dEQP
459 test case as a separate 'frame', just for the purpose of capturing. The frames
460 are added using RenderDoc 'In-Application API', instead of swapchain operations.