4 This document describes how to build and run Vulkan Conformance Test suite.
6 Vulkan CTS is built on dEQP framework. dEQP documentation is available
7 at http://source.android.com/devices/graphics/testing.html
15 The following tools must be installed and present in the PATH variable:
17 * Git (for checking out sources)
18 * Python 3.x (for the build related scripts, some other scripts still use Python 2.7.x)
19 * CMake 3.10.2 or newer
23 * Visual Studio 2015 or newer (glslang uses several C++11 features)
27 * Standard toolchain (make, gcc/clang)
28 * If you have X11 installed, then the build assumes you also have the `GL/glx.h` header
30 * You can get this from the `mesa-common-dev` Ubuntu package.
36 Vulkan SDK for macOS includes a MoltenVK runtime library which is recommended method for
37 developing a Vulkan application.
41 * Android NDK r17c or later.
42 * Android SDK with: SDK Tools, SDK Platform-tools, SDK Build-tools, and API 28
43 * Java Development Kit (JDK)
44 * Windows: either NMake or Ninja in PATH
46 If you have downloaded the Android SDK command line tools package (25.2.3 or higher) then
47 you can install the necessary components by running:
49 tools/bin/sdkmanager tools platform-tools 'build-tools;25.0.2' 'platforms;android-28'
55 To build dEQP, you need first to download sources for zlib, libpng, glslang,
56 vulkan-docs, spirv-headers, and spirv-tools.
58 To download sources, run:
60 python external/fetch_sources.py
62 You may need to re-run `fetch_sources.py` to update to the latest glslang,
63 vulkan-docs and spirv-tools revisions occasionally.
65 You also need to install lxml python module by running:
67 python -m pip install lxml
69 With CMake out-of-source builds are always recommended. Create a build directory
70 of your choosing, and in that directory generate Makefiles or IDE project
76 cmake <path to vulkancts> -G"Visual Studio 14"
77 start dEQP-Core-default.sln
81 cmake <path to vulkancts> -G"Visual Studio 14 Win64"
82 start dEQP-Core-default.sln
84 ### Linux 32-bit Debug
86 cmake <path to vulkancts> -DCMAKE_BUILD_TYPE=Debug -DCMAKE_C_FLAGS=-m32 -DCMAKE_CXX_FLAGS=-m32
89 Release build can be done by using -DCMAKE_BUILD_TYPE=Release
91 If building for 32-bit x86 with GCC, you probably also want to add `-msse2
92 -mfpmath=sse` to ensure that you get correct IEEE floating-point behavior.
94 ### Linux 64-bit Debug
96 cmake <path to vulkancts> -DCMAKE_BUILD_TYPE=Debug -DCMAKE_C_FLAGS=-m64 -DCMAKE_CXX_FLAGS=-m64
100 ### MacOS 64-bit Debug
102 cmake <path to vulkancts> -DCMAKE_BUILD_TYPE=Debug -DDEQP_TARGET=osx -DCMAKE_C_FLAGS=-m64 -DCMAKE_CXX_FLAGS=-m64
107 Following command will build dEQP.apk:
109 python scripts/android/build_apk.py --sdk <path to Android SDK> --ndk <path to Android NDK>
111 By default the CTS package will be built for the Android API level 28.
112 Another API level may be supplied using --native-api command line option.
114 The package can be installed by either running:
116 python scripts/android/install_apk.py
118 By default the CTS package will contain libdeqp.so built for armeabi-v7a, arm64-v8a,
119 x86, and x86_64 ABIs, but that can be changed at build time by passing the --abis command line
120 option to `scripts/android/build_apk.py`.
122 To pick which ABI to use at _install time_, use the following command instead:
124 adb install -g --abi <ABI name> <build-root>/package/dEQP.apk
130 Current mustpass is checked into repository and can be found at:
132 external/vulkancts/mustpass/master/vk-default.txt
134 This file contains list of files located in vk-default subdirectory. Those files contain
135 tests from top-level groups and together they contain all test cases that should pass.
137 Vulkan CTS mustpass can be re-generated by running:
139 python <vulkancts>/external/vulkancts/scripts/build_mustpass.py
145 Following command line options MUST be used when running CTS:
147 --deqp-caselist-file=<vulkancts>/external/vulkancts/mustpass/master/vk-default.txt
148 --deqp-log-images=disable
149 --deqp-log-shader-sources=disable
151 In addition on multi-device systems the device for which conformance is claimed
152 can be selected with:
154 --deqp-vk-device-id=<value>
156 To speed up the conformance run on some platforms the following command line
157 option may be used to disable frequent fflush() calls to the output logs:
159 --deqp-log-flush=disable
161 By default, the test log will be written into the path "TestResults.qpa". If the
162 platform requires a different path, it can be specified with:
164 --deqp-log-filename=<path>
166 By default, the CTS will expect to find its test resource files in the current
167 working directory. This can be overridden with:
169 --deqp-archive-dir=<path>
171 By default, the shader cache will be written into the path "shadercache.bin". If the
172 platform requires a different path, it can be specified with:
174 --deqp-shadercache-filename=<path>
176 If the shader cache is not desired, it can be disabled with:
178 --deqp-shadercache=disable
180 CTS execution may be split into N fractions ( for the purpose of running it in parallel ) using
184 where I denotes index of current CTS execution ( I=[0..N-1], N=[1..8] )
186 When collecting results for a Conformance Submission Package the number of fractions must not exceed 8,
187 and a list of mandatory information tests for each fraction must be supplied:
189 --deqp-fraction-mandatory-caselist-file=<vulkancts>external/vulkancts/mustpass/master/vk-fraction-mandatory-tests.txt
191 To specify file containing waived tests that are omitted only by specified vendor and renderer/device
192 the following command line option may be used:
194 --deqp-waiver-file=<path>
196 To make log more clear for high level shader source analysis
197 output of decompiled SPIR-V shader sources can be disabled:
199 --deqp-log-decompiled-spirv=disable
201 Also non-informative empty LogInfo sections can be removed
202 from output into log by specifying:
204 --deqp-log-empty-loginfo=disable
206 No other command line options are allowed.
210 cd <builddir>/external/vulkancts/modules/vulkan
211 Debug\deqp-vk.exe --deqp-caselist-file=...
213 Test log will be written into TestResults.qpa
217 cd <builddir>/external/vulkancts/modules/vulkan
218 ./deqp-vk --deqp-caselist-file=...
222 cd <builddir>/external/vulkancts/modules/vulkan
223 ./deqp-vk --deqp-caselist-file=...
227 adb push <vulkancts>/external/vulkancts/mustpass/master/vk-default.txt /sdcard/vk-default.txt
232 am start -n com.drawelements.deqp/android.app.NativeActivity -e cmdLine "deqp --deqp-caselist-file=/sdcard/vk-default.txt --deqp-log-images=disable --deqp-log-shader-sources=disable --deqp-log-filename=/sdcard/TestResults.qpa"
234 Test progress will be written to device log and can be displayed with:
238 Test log will be written into `/sdcard/TestResults.qpa`.
241 Conformance Submission Package Requirements
242 -------------------------------------------
244 The conformance submission package must contain the following:
246 1. Full test logs (`TestResults.qpa`) from CTS runs against all driver builds and all fractions
247 2. Result of `git status` and `git log` from CTS source directory
248 3. Any patches used on top of release tag
249 4. Conformance statement
251 Test logs (1) should be named `<submission pkg dir>/TestResults-<driver build type>-<fraction id>-of-<total fractions>.qpa`,
252 for example `TestResults-armeabi-v7a-1-of-8.qpa`. On platforms where multiple different driver
253 builds (for example 64-bit and 32-bit) are present, CTS logs must be provided
254 for each driver build as part of the submission package. If CTS run was split into multiple
255 fractions then result files for all fractions must be provided, each file must
256 contain results of the mandatory information tests.
258 Fractions may be run on different physical devices but each device must represent
259 the same Conformant Product.
261 Test logs generated on a system which exposes more than one physical device
262 in a device group can be used for products that expose one or more physical
263 devices in their device group.
265 The CTS build must always be done from clean git repository that doesn't have any
266 uncommitted changes. Thus it is necessary to run and capture output of `git
267 status` and `git log` (2) in the source directory:
269 git status > <submission pkg dir>/git-status.txt
270 git log --first-parent <release tag>^..HEAD > <submission pkg dir>/git-log.txt
272 Any changes made to CTS must be committed to the local repository, and provided
273 as part of the submission package (3). This can be done by running:
275 git format-patch -o <submission pkg dir> <release tag>..HEAD
277 Changes to platform-specific code (mostly under `framework/platform`)
278 are allowed. The commit message for each change must include a clear
279 description of the change and why it is necessary.
281 Bugfixes to the tests are allowed. Before being used for a submission,
282 bugfixes must be accepted and merged into the CTS repository.
283 `git cherry-pick` is strongly recommended as a method of applying bug fixes.
285 Other changes must be accompanied by a waiver (see below).
287 NOTE: When cherry-picking patches on top of release tag, please use `git cherry-pick -x`
288 to include original commit hash in the commit message.
290 Conformance statement (4) must be included in a file called `STATEMENT-<adopter>`
291 and must contain following:
293 CONFORM_VERSION: <git tag of CTS release>
294 PRODUCT: <string-value>
298 Note that product/cpu/os information is also captured in `dEQP-VK.info.*` tests
299 if `vk::Platform::describePlatform()` is implemented.
301 If the submission package covers multiple products, you can list them by appending
302 additional `PRODUCT:` lines to the conformance statement. For example:
304 CONFORM_VERSION: vulkan-cts-1.2.6.0
309 The actual submission package consists of the above set of files which must
310 be bundled into a gzipped tar file named `VK<API major><API minor>_<adopter><_info>.tgz`.
311 `<API major>` is the major version of the Vulkan API specification, `<API minor>`is the minor
312 version of the Vulkan API specification.
313 `<adopter>` is the name of the Adopting member company, or some recognizable abbreviation.
314 The `<_info>` field is optional. It may be used to uniquely identify a submission
315 by OS, platform, date, or other criteria when making multiple submissions.
316 For example, a company XYZ may make a submission for a Vulkan 1.1 implementation named
317 `VK11_XYZ_PRODUCTA_Windows10.tgz`
319 One way to create a suiteable gzipped tar file is to execute the command:
321 tar -cvzf <filename.tgz> -C <submission pkg dir> .
323 where `<submission pkg dir>` is the directory containing the files from (1)-(4)
324 from above. A submission package must contain all of the files listed above,
325 and only those files.
327 As an example submission package could contain:
332 0001-Remove-Waived-Filtering-Tests.patch
333 0002-Fix-Pipeline-Parameters.patch
334 TestResults-armeabi-v7a.qpa
335 TestResults-arm64-v8a.qpa
341 The process for requesting a waiver is to report the issue by filing a bug
342 report in the Gitlab VulkanCTS project (TODO Github?). When creating the
343 submission package, include references to the waiver in the commit message of
344 the relevant change. Including as much information as possible in your bug
345 report (including a unified diff or a merge request of suggested file changes)
346 will ensure the issue can be progressed as rapidly as possible. Issues must
347 be labeled "Waiver" (TODO!) and identify the version of the CTS and affected
353 Conformance run is considered passing if all tests finish with allowed result
354 codes. Test results are contained in the TestResults.qpa log. Each
355 test case section contains XML tag Result, for example:
357 <Result StatusCode="Pass">Not validated</Result>
359 The result code is the value of the StatusCode attribute. Following status
368 Submission package can be verified using `verify_submission.py`
369 script located in [VK-GL-CTS-Tools](https://github.com/KhronosGroup/VK-GL-CTS-Tools).
374 Vulkan support from Platform implementation requires providing
375 `getVulkanPlatform()` method in `tcu::Platform` class implementation.
377 See `framework/common/tcuPlatform.hpp` and examples in
378 `framework/platform/win32/tcuWin32Platform.cpp` and
379 `framework/platform/android/tcuAndroidPlatform.cpp`.
381 If any WSI extensions are supported, platform port must also implement
382 methods for creating native display (`vk::Platform::createWsiDisplay`)
383 and window handles (`vk::wsi::Display::createWindow`). Otherwise tests
384 under `dEQP-VK.wsi` will fail.
390 For testing and development purposes it might be useful to be able to run
391 tests on dummy Vulkan implementation. One such implementation is provided in
392 vkNullDriver.cpp. To use that, implement `vk::Platform::createLibrary()` with
393 `vk::createNullDriver()`.
399 Vulkan CTS framework includes first-party support for validation layers, that
400 can be turned on with `--deqp-validation=enable` command line option.
402 When validation is turned on, default instance and device will be created with
403 validation layers enabled and debug callback is registered to record any
404 messages. Debug messages collected during test execution will be included at
405 the end of the test case log.
407 In addition, when the `--deqp-print-validation-errors` command line option is
408 used, validation errors are additionally printed to standard error in the
409 moment they are generated.
411 If any validation errors are found, test result will be set to `InternalError`.
413 By default `VK_DEBUG_REPORT_INFORMATION_BIT_EXT` and `_DEBUG_BIT_EXT` messages
414 are excluded from the log, but that can be customized by modifying
415 `vk::DebugReportMessage::shouldBeLogged()` in `vkDebugReportUtil.hpp`.
417 On the Android target, layers can be added to the APK during the build process
418 by setting the `--layers-path` command line option to point into the NDK or to
419 a locally-built layers tree. The layers are expected to be found under $abi/
420 under the layers path.
426 Vulkan test module can be used with Cherry (GUI for test execution and
427 analysis). Cherry is available at
428 https://android.googlesource.com/platform/external/cherry. Please follow
429 instructions in README to get started.
431 Before first launch, and every time test hierarchy has been modified, test
432 case list must be refreshed by running:
434 python scripts/build_caselists.py path/to/cherry/data
436 Cherry must be restarted for the case list update to take effect.
442 Vulkan CTS can be optionally run with the shader optimizer enabled. This
443 is an experimental feature which can be used to further stress both the
444 drivers as well as the optimizer itself. The shader optimizer is disabled
447 The following command line options can be used to configure the shader
450 --deqp-optimization-recipe=<number>
452 The list of the optimization recipes can be found and customized in the
453 `optimizeCompiledBinary()` function in `vkPrograms.cpp`.
455 As of this writing, there are 2 recipes to choose from:
457 0. Disabled (default)
458 1. Optimize for performance
461 The performance and size optimization recipes are defined by the spir-v
462 optimizer, and will change from time to time as the optimizer matures.
464 --deqp-optimize-spirv=enable
466 This option is not required to run the optimizer. By default, the shader
467 optimizer only optimizes shaders generated from GLSL or HLSL, and leaves
468 hand-written SPIR-V shaders alone.
470 Many of the hand-written SPIR-V tests stress specific features of the
471 SPIR-V which might get optimized out. Using this option will enable the
472 optimizer on the hand-written SPIR-V as well, which may be useful in
473 finding new bugs in drivers or the optimizer itself, but will likely
474 invalidate the tests themselves.
480 The Vulkan CTS framework contains a shader cache for speeding up the running
481 of the CTS. Skipping shader compilation can significantly reduce runtime,
482 especially for repeated runs.
484 Default behavior is to have the shader cache enabled, but truncated at the
485 start of the CTS run. This still gives the benefit of skipping shader
486 compilation for identical shaders in different tests (which there are many),
487 while making sure that the shader cache file does not grow indefinitely.
489 The shader cache identifies the shaders by hashing the shader source code
490 along with various bits of information that may affect the shader compilation
491 (such as shader stage, CTS version, possible compilation flags, etc). If a
492 cached shader with matching hash is found, a byte-by-byte comparison of the
493 shader sources is made to make sure that the correct shader is being
494 retrieved from the cache.
496 The behavior of the shader cache can be modified with the following command
499 --deqp-shadercache=disable
501 Disable the shader cache. All shaders will be compiled every time.
503 --deqp-shadercache-filename=<filename>
505 Set the name of the file where the cached shaders will be stored. This
506 option may be required for the shader cache to work at all on Android
509 --deqp-shadercache-truncate=disable
511 Do not truncate the shader cache file at startup. No shader compilation will
512 occur on repeated runs of the CTS.
517 The RenderDoc (https://renderdoc.org/) graphics debugger may be used to debug
520 Following command line option should be used when launching tests from
523 --deqp-renderdoc=enable
525 This causes the framework to interface with the debugger and mark each dEQP
526 test case as a separate 'frame', just for the purpose of capturing. The frames
527 are added using RenderDoc 'In-Application API', instead of swapchain operations.
533 Some CTS tests use third-party runners. By default all tests are executed
534 regardless of runner type (`any`). To exclude all tests using any of the
535 external runners (`none`) or to only include tests using a certain runner:
537 --deqp-runner-type=(any|none|amber)
542 Full list of parameters for the `deqp-vk` module:
544 OpenGL and OpenCL parameters not affecting Vulkan API were suppressed.
549 -n, --deqp-case=<value>
550 Test case(s) to run, supports wildcards (e.g. dEQP-GLES2.info.*)
552 --deqp-caselist=<value>
553 Case list to run in trie format (e.g. {dEQP-GLES2{info{version,renderer}}})
555 --deqp-caselist-file=<value>
556 Read case list (in trie format) from given file
558 --deqp-caselist-resource=<value>
559 Read case list (in trie format) from given file located application's assets
561 --deqp-stdin-caselist
562 Read case list (in trie format) from stdin
564 --deqp-log-filename=<value>
565 Write test results to given file
566 default: 'TestResults.qpa'
568 --deqp-runmode=[execute|xml-caselist|txt-caselist|stdout-caselist]
569 Execute tests, or write list of test cases into a file
572 --deqp-caselist-export-file=<value>
573 Set the target file name pattern for caselist export
574 default: '${packageName}-cases.${typeExtension}'
576 --deqp-watchdog=[enable|disable]
580 --deqp-crashhandler=[enable|disable]
581 Enable crash handling
584 --deqp-base-seed=<value>
585 Base seed for test cases that use randomization
588 --deqp-test-iteration-count=<value>
589 Iteration count for cases that support variable number of iterations
592 --deqp-visibility=[windowed|fullscreen|hidden]
593 Default test window visibility
596 --deqp-surface-width=<value>
597 Use given surface width if possible
600 --deqp-surface-height=<value>
601 Use given surface height if possible
604 --deqp-surface-type=[window|pixmap|pbuffer|fbo]
605 Use given surface type
608 --deqp-screen-rotation=[unspecified|0|90|180|270]
609 Screen rotation for platforms that support it
612 --deqp-vk-device-id=<value>
613 Vulkan device ID (IDs start from 1)
616 --deqp-vk-device-group-id=<value>
617 Vulkan device Group ID (IDs start from 1)
620 --deqp-log-images=[enable|disable]
621 Enable or disable logging of result images
624 --deqp-log-shader-sources=[enable|disable]
625 Enable or disable logging of shader sources
628 --deqp-test-oom=[enable|disable]
629 Run tests that exhaust memory on purpose
632 --deqp-archive-dir=<value>
633 Path to test resource files
636 --deqp-log-flush=[enable|disable]
637 Enable or disable log file fflush
640 --deqp-validation=[enable|disable]
641 Enable or disable test case validation
644 --deqp-print-validation-errors
645 Print validation errors to standard error
647 --deqp-optimization-recipe=<value>
648 Shader optimization recipe (0=disabled, 1=performance, 2=size)
651 --deqp-optimize-spirv=[enable|disable]
652 Apply optimization to spir-v shaders as well
655 --deqp-shadercache=[enable|disable]
656 Enable or disable shader cache
659 --deqp-shadercache-filename=<value>
660 Write shader cache to given file
661 default: 'shadercache.bin'
663 --deqp-shadercache-truncate=[enable|disable]
664 Truncate shader cache before running tests
667 --deqp-renderdoc=[enable|disable]
668 Enable RenderDoc frame markers
671 --deqp-fraction=<value>
672 Run a fraction of the test cases (e.g. N,M means run group%M==N)
675 --deqp-fraction-mandatory-caselist-file=<value>
676 Case list file that must be run for each fraction
679 --deqp-waiver-file=<value>
680 Read waived tests from given file
683 --deqp-runner-type=[any|none|amber]
684 Filter test cases based on runner
687 --deqp-terminate-on-fail=[enable|disable]
688 Terminate the run on first failure