4 This document describes how to build and run Vulkan Conformance Test suite.
6 Vulkan CTS is built on dEQP framework. dEQP documentation is available
7 at http://source.android.com/devices/graphics/testing.html
15 The following tools must be installed and present in the PATH variable:
17 * Git (for checking out sources)
18 * Python 3.x (for the build related scripts, some other scripts still use Python 2.7.x)
19 * CMake 3.0 (3.6 for Android NDK r17+ builds) or newer
23 * Visual Studio 2015 or newer (glslang uses several C++11 features)
27 * Standard toolchain (make, gcc/clang)
31 * Android NDK r15c or later.
32 * Android SDK with: SDK Tools, SDK Platform-tools, SDK Build-tools, and API 28
33 * Java Development Kit (JDK)
34 * Windows: either NMake or Ninja in PATH
36 If you have downloaded Android SDK tools, you can install necessary components
39 tools/android update sdk --no-ui --all --filter tools,platform-tools,build-tools-25.0.2,android-28
45 To build dEQP, you need first to download sources for zlib, libpng, glslang,
46 spirv-headers, and spirv-tools.
48 To download sources, run:
50 python external/fetch_sources.py
52 You may need to re-run `fetch_sources.py` to update to the latest glslang and
53 spirv-tools revisions occasionally.
55 With CMake out-of-source builds are always recommended. Create a build directory
56 of your choosing, and in that directory generate Makefiles or IDE project
62 cmake <path to vulkancts> -G"Visual Studio 14"
63 start dEQP-Core-default.sln
67 cmake <path to vulkancts> -G"Visual Studio 14 Win64"
68 start dEQP-Core-default.sln
70 ### Linux 32-bit Debug
72 cmake <path to vulkancts> -DCMAKE_BUILD_TYPE=Debug -DCMAKE_C_FLAGS=-m32 -DCMAKE_CXX_FLAGS=-m32
75 Release build can be done by using -DCMAKE_BUILD_TYPE=Release
77 If building for 32-bit x86 with GCC, you probably also want to add `-msse2
78 -mfpmath=sse` to ensure that you get correct IEEE floating-point behavior.
80 ### Linux 64-bit Debug
82 cmake <path to vulkancts> -DCMAKE_BUILD_TYPE=Debug -DCMAKE_C_FLAGS=-m64 -DCMAKE_CXX_FLAGS=-m64
87 Following command will build dEQP.apk:
89 python scripts/android/build_apk.py --sdk <path to Android SDK> --ndk <path to Android NDK>
91 By default the CTS package will be built for the Android API level 28.
92 Another API level may be supplied using --native-api command line option.
94 The package can be installed by either running:
96 python scripts/android/install_apk.py
98 By default the CTS package will contain libdeqp.so built for armeabi-v7a, arm64-v8a,
99 x86, and x86_64 ABIs, but that can be changed using --abis command line option.
101 To pick which ABI to use at install time, following commands must be used
104 adb install --abi <ABI name> <build-root>/package/dEQP.apk /data/local/tmp/dEQP-debug.apk
110 Current mustpass is checked into repository and can be found at:
112 external/vulkancts/mustpass/master/vk-default.txt
114 Vulkan CTS mustpass can be re-generated by running:
116 python <vulkancts>/external/vulkancts/scripts/build_mustpass.py
119 Pre-compiling SPIR-V binaries
120 -----------------------------
122 For distribution, and platforms that don't support GLSL to SPIR-V compilation,
123 SPIR-V binaries can be pre-built with following command:
125 python external/vulkancts/scripts/build_spirv_binaries.py
127 By default the script builds SPIR-V binaries for Vulkan 1.1.
128 Binaries for other Vulkan versions can be requested by supplying
129 an extra command line option:
131 python external/vulkancts/scripts/build_spirv_binaries.py --target-vulkan-version <Vulkan version>
133 Binaries will be written to `external/vulkancts/data/vulkan/prebuilt/`.
135 Test modules (or in case of Android, the APK) must be re-built after building
136 SPIR-V programs in order for the binaries to be available.
142 Following command line options MUST be used when running CTS:
144 --deqp-caselist-file=<vulkancts>/external/vulkancts/mustpass/master/vk-default.txt
145 --deqp-log-images=disable
146 --deqp-log-shader-sources=disable
148 In addition on multi-device systems the device for which conformance is claimed
149 can be selected with:
151 --deqp-vk-device-id=<value>
153 To speed up the conformance run on some platforms the following command line
154 option may be used to disable frequent fflush() calls to the output logs:
156 --deqp-log-flush=disable
158 By default, the test log will be written into the path "TestResults.qpa". If the
159 platform requires a different path, it can be specified with:
161 --deqp-log-filename=<path>
163 By default, the shader cache will be written into the path "shadercache.bin". If the
164 platform requires a different path, it can be specified with:
166 --deqp-shadercache-filename=<path>
168 If the shader cache is not desired, it can be disabled with:
170 --deqp-shadercache=disable
172 No other command line options are allowed.
176 cd <builddir>/external/vulkancts/modules/vulkan
177 Debug\deqp-vk.exe --deqp-caselist-file=...
179 Test log will be written into TestResults.qpa
183 cd <builddir>/external/vulkancts/modules/vulkan
184 ./deqp-vk --deqp-vk-caselist-file=...
188 adb push <vulkancts>/external/vulkancts/mustpass/master/vk-default.txt /sdcard/vk-default.txt
193 am start -n com.drawelements.deqp/android.app.NativeActivity -e cmdLine "deqp --deqp-caselist-file=/sdcard/vk-default.txt --deqp-log-images=disable --deqp-log-shader-sources=disable --deqp-log-filename=/sdcard/TestResults.qpa"
195 Test progress will be written to device log and can be displayed with:
199 Test log will be written into `/sdcard/TestResults.qpa`.
202 Conformance Submission Package Requirements
203 -------------------------------------------
205 The conformance submission package must contain the following:
207 1. Full test logs (`TestResults.qpa`) from CTS runs against all driver builds
208 2. Result of `git status` and `git log` from CTS source directory
209 3. Any patches used on top of release tag
210 4. Conformance statement
212 Test logs (1) should be named `<submission pkg dir>/TestResults-<driver build type>.qpa`,
213 for example `TestResults-armeabi-v7a.qpa`. On platforms where multiple different driver
214 builds (for example 64-bit and 32-bit) are present, CTS logs must be provided
215 for each driver build as part of the submission package.
217 Test logs generated on a system which exposes more than one physical device
218 in a device group can be used for products that expose one or more physical
219 devices in their device group.
221 The CTS build must always be done from clean git repository that doesn't have any
222 uncommitted changes. Thus it is necessary to run and capture output of `git
223 status` and `git log` (2) in the source directory:
225 git status > <submission pkg dir>/git-status.txt
226 git log --first-parent <release tag>^..HEAD > <submission pkg dir>/git-log.txt
228 Any changes made to CTS must be committed to the local repository, and provided
229 as part of the submission package (3). This can be done by running:
231 git format-patch -o <submission pkg dir> <release tag>..HEAD
233 In general, bugfixes and changes to platform-specific code (mostly under
234 `framework/platform`) are allowed. The commit message for each change must
235 include a clear description of the change and why it is necessary. Non-porting
236 related changes must be accompanied by a waiver (see below).
238 NOTE: When cherry-picking patches on top of release tag, please use `git cherry-pick -x`
239 to include original commit hash in the commit message.
241 Conformance statement (4) must be included in a file called `STATEMENT-<adopter>`
242 and must contain following:
244 CONFORM_VERSION: <git tag of CTS release>
245 PRODUCT: <string-value>
249 Note that product/cpu/os information is also captured in `dEQP-VK.info.*` tests
250 if `vk::Platform::describePlatform()` is implemented.
252 If the submission package covers multiple products, you can list them by appending
253 additional `PRODUCT:` lines to the conformance statement. For example:
255 CONFORM_VERSION: vulkan-cts-1.1.5.0
260 The actual submission package consists of the above set of files which must
261 be bundled into a gzipped tar file named `VK<API major><API minor>_<adopter><_info>.tgz`.
262 `<API major>` is the major version of the Vulkan API specification, `<API minor>`is the minor
263 version of the Vulkan API specification.
264 `<adopter>` is the name of the Adopting member company, or some recognizable abbreviation.
265 The `<_info>` field is optional. It may be used to uniquely identify a submission
266 by OS, platform, date, or other criteria when making multiple submissions.
267 For example, a company XYZ may make a submission for a Vulkan 1.1 implementation named
268 `VK11_XYZ_PRODUCTA_Windows10.tgz`
270 One way to create a suiteable gzipped tar file is to execute the command:
272 tar -cvzf <filename.tgz> -C <submission pkg dir> .
274 where `<submission pkg dir>` is the directory containing the files from (1)-(4)
275 from above. A submission package must contain all of the files listed above,
276 and only those files.
278 As an example submission package could contain:
283 0001-Remove-Waived-Filtering-Tests.patch
284 0002-Fix-Pipeline-Parameters.patch
285 TestResults-armeabi-v7a.qpa
286 TestResults-arm64-v8a.qpa
292 The process for requesting a waiver is to report the issue by filing a bug
293 report in the Gitlab VulkanCTS project (TODO Github?). When creating the
294 submission package, include references to the waiver in the commit message of
295 the relevant change. Including as much information as possible in your bug
296 report (including a unified diff or a merge request of suggested file changes)
297 will ensure the issue can be progressed as rapidly as possible. Issues must
298 be labeled "Waiver" (TODO!) and identify the version of the CTS and affected
304 Conformance run is considered passing if all tests finish with allowed result
305 codes. Test results are contained in the TestResults.qpa log. Each
306 test case section contains XML tag Result, for example:
308 <Result StatusCode="Pass">Not validated</Result>
310 The result code is the value of the StatusCode attribute. Following status
318 Submission package can be verified using `external/vulkancts/scripts/verify_submission.py`
319 script. The script takes two arguments: path to extracted submission package
320 and path to current mustpass list. For example:
322 python external/vulkancts/scripts/verify_submission.py VK_11_Khronos_1/ external/vulkancts/mustpass/master/vk-default.txt
324 Please note that the script reports a warning even for a correctly generated git-log.txt
325 If your git-log.txt contains only head commit of the release tag then
326 the warning can be ignored.
331 Vulkan support from Platform implementation requires providing
332 `getVulkanPlatform()` method in `tcu::Platform` class implementation.
334 See `framework/common/tcuPlatform.hpp` and examples in
335 `framework/platform/win32/tcuWin32Platform.cpp` and
336 `framework/platform/android/tcuAndroidPlatform.cpp`.
338 If any WSI extensions are supported, platform port must also implement
339 methods for creating native display (`vk::Platform::createWsiDisplay`)
340 and window handles (`vk::wsi::Display::createWindow`). Otherwise tests
341 under `dEQP-VK.wsi` will fail.
347 For testing and development purposes it might be useful to be able to run
348 tests on dummy Vulkan implementation. One such implementation is provided in
349 vkNullDriver.cpp. To use that, implement `vk::Platform::createLibrary()` with
350 `vk::createNullDriver()`.
356 Vulkan CTS framework includes first-party support for validation layers, that
357 can be turned on with `--deqp-validation=enable` command line option.
359 When validation is turned on, default instance and device will be created with
360 validation layers enabled and debug callback is registered to record any
361 messages. Debug messages collected during test execution will be included at
362 the end of the test case log.
364 If any validation errors are found, test result will be set to `InternalError`.
366 By default `VK_DEBUG_REPORT_INFORMATION_BIT_EXT` and `_DEBUG_BIT_EXT` messages
367 are excluded from the log, but that can be customized by modifying
368 `vkt::TestCaseExecutor::deinit()` in `vktTestPackage.cpp`.
370 On the Android target, layers can be added to the APK during the build process
371 by setting the `--layers-path` command line option to point into the NDK or to
372 a locally-built layers tree. The layers are expected to be found under $abi/
373 under the layers path.
379 Vulkan test module can be used with Cherry (GUI for test execution and
380 analysis). Cherry is available at
381 https://android.googlesource.com/platform/external/cherry. Please follow
382 instructions in README to get started.
384 Before first launch, and every time test hierarchy has been modified, test
385 case list must be refreshed by running:
387 python scripts/build_caselists.py path/to/cherry/data
389 Cherry must be restarted for the case list update to take effect.
395 Vulkan CTS can be optionally run with the shader optimizer enabled. This
396 is an experimental feature which can be used to further stress both the
397 drivers as well as the optimizer itself. The shader optimizer is disabled
400 The following command line options can be used to configure the shader
403 --deqp-optimization-recipe=<number>
405 The list of the optimization recipes can be found and customized in the
406 `optimizeCompiledBinary()` function in `vkPrograms.cpp`.
408 As of this writing, there are 2 recipes to choose from:
410 0. Disabled (default)
411 1. Optimize for performance
414 The performance and size optimization recipes are defined by the spir-v
415 optimizer, and will change from time to time as the optimizer matures.
417 --deqp-optimize-spirv=enable
419 This option is not required to run the optimizer. By default, the shader
420 optimizer only optimizes shaders generated from GLSL or HLSL, and leaves
421 hand-written SPIR-V shaders alone.
423 Many of the hand-written SPIR-V tests stress specific features of the
424 SPIR-V which might get optimized out. Using this option will enable the
425 optimizer on the hand-written SPIR-V as well, which may be useful in
426 finding new bugs in drivers or the optimizer itself, but will likely
427 invalidate the tests themselves.
433 The Vulkan CTS framework contains a shader cache for speeding up the running
434 of the CTS. Skipping shader compilation can significantly reduce runtime,
435 especially for repeated runs.
437 Default behavior is to have the shader cache enabled, but truncated at the
438 start of the CTS run. This still gives the benefit of skipping shader
439 compilation for identical shaders in different tests (which there are many),
440 while making sure that the shader cache file does not grow indefinitely.
442 The shader cache identifies the shaders by hashing the shader source code
443 along with various bits of information that may affect the shader compilation
444 (such as shader stage, CTS version, possible compilation flags, etc). If a
445 cached shader with matching hash is found, a byte-by-byte comparison of the
446 shader sources is made to make sure that the correct shader is being
447 retrieved from the cache.
449 The behavior of the shader cache can be modified with the following command
452 --deqp-shadercache=disable
454 Disable the shader cache. All shaders will be compiled every time.
456 --deqp-shadercache-filename=<filename>
458 Set the name of the file where the cached shaders will be stored. This
459 option may be required for the shader cache to work at all on Android
462 --deqp-shadercache-truncate=disable
464 Do not truncate the shader cache file at startup. No shader compilation will
465 occur on repeated runs of the CTS.
470 The RenderDoc (https://renderdoc.org/) graphics debugger may be used to debug
473 Following command line option should be used when launching tests from
476 --deqp-renderdoc=enable
478 This causes the framework to interface with the debugger and mark each dEQP
479 test case as a separate 'frame', just for the purpose of capturing. The frames
480 are added using RenderDoc 'In-Application API', instead of swapchain operations.