Merge vk-gl-cts/vulkan-cts-1.1.3 into vk-gl-cts/vulkan-cts-1.1.4

[platform/upstream/VK-GL-CTS.git] / external / vulkancts / README.md

Vulkan CTS README
=================

This document describes how to build and run Vulkan Conformance Test suite.

Vulkan CTS is built on dEQP framework. dEQP documentation is available
at http://source.android.com/devices/graphics/testing.html


Requirements
------------

### Common

The following tools must be installed and present in the PATH variable:

 * Git (for checking out sources)
 * Python 3.x (for the build related scripts, some other scripts still use Python 2.7.x)
 * CMake 2.8 (3.6 for Android NDK r17+ builds) or newer

### Win32

 * Visual Studio 2015 or newer (glslang uses several C++11 features)

### Linux

 * Standard toolchain (make, gcc/clang)

### Android

 * Android NDK r15c or later.
 * Android SDK with: SDK Tools, SDK Platform-tools, SDK Build-tools, and API 28
 * Java Development Kit (JDK)
 * Windows: either NMake or Ninja in PATH

If you have downloaded Android SDK tools, you can install necessary components
by running:

        tools/android update sdk --no-ui --all --filter tools,platform-tools,build-tools-25.0.2,android-28


Building CTS
------------

To build dEQP, you need first to download sources for zlib, libpng, glslang,
spirv-headers, and spirv-tools.

To download sources, run:

        python external/fetch_sources.py

You may need to re-run `fetch_sources.py` to update to the latest glslang and
spirv-tools revisions occasionally.

With CMake out-of-source builds are always recommended. Create a build directory
of your choosing, and in that directory generate Makefiles or IDE project
using cmake.


### Windows x86-32

        cmake <path to vulkancts> -G"Visual Studio 14"
        start dEQP-Core-default.sln

### Windows x86-64

        cmake <path to vulkancts> -G"Visual Studio 14 Win64"
        start dEQP-Core-default.sln

### Linux 32-bit Debug

        cmake <path to vulkancts> -DCMAKE_BUILD_TYPE=Debug -DCMAKE_C_FLAGS=-m32 -DCMAKE_CXX_FLAGS=-m32
        make -j

Release build can be done by using -DCMAKE_BUILD_TYPE=Release

### Linux 64-bit Debug

        cmake <path to vulkancts> -DCMAKE_BUILD_TYPE=Debug -DCMAKE_C_FLAGS=-m64 -DCMAKE_CXX_FLAGS=-m64
        make -j

### Android

Following command will build dEQP.apk:

        python scripts/android/build_apk.py --sdk <path to Android SDK> --ndk <path to Android NDK>

By default the CTS package will be built for the Android API level 28.
Another API level may be supplied using --native-api command line option.

The package can be installed by either running:

        python scripts/android/install_apk.py

By default the CTS package will contain libdeqp.so built for armeabi-v7a, arm64-v8a,
x86, and x86_64 ABIs, but that can be changed using --abis command line option.

To pick which ABI to use at install time, following commands must be used
instead:

        adb install --abi <ABI name> <build-root>/package/dEQP.apk /data/local/tmp/dEQP-debug.apk


Building Mustpass
-----------------

Current mustpass is checked into repository and can be found at:

        external/vulkancts/mustpass/1.1.4/vk-default.txt

Vulkan CTS mustpass can be re-generated by running:

        python <vulkancts>/external/vulkancts/scripts/build_mustpass.py


Pre-compiling SPIR-V binaries
-----------------------------

For distribution, and platforms that don't support GLSL to SPIR-V compilation,
SPIR-V binaries can be pre-built with following command:

        python external/vulkancts/scripts/build_spirv_binaries.py

By default the script builds SPIR-V binaries for Vulkan 1.1.
Binaries for other Vulkan versions can be requested by supplying
an extra command line option:

        python external/vulkancts/scripts/build_spirv_binaries.py --target-vulkan-version <Vulkan version>

Binaries will be written to `external/vulkancts/data/vulkan/prebuilt/`.

Test modules (or in case of Android, the APK) must be re-built after building
SPIR-V programs in order for the binaries to be available.


Running CTS
-----------

Following command line options MUST be used when running CTS:

        --deqp-caselist-file=<vulkancts>/external/vulkancts/mustpass/1.1.4/vk-default.txt
        --deqp-log-images=disable
        --deqp-log-shader-sources=disable

In addition on multi-device systems the device for which conformance is claimed
can be selected with:

        --deqp-vk-device-id=<value>

To speed up the conformance run on some platforms the following command line
option may be used to disable frequent fflush() calls to the output logs:

        --deqp-log-flush=disable

By default, the test log will be written into the path "TestResults.qpa". If the
platform requires a different path, it can be specified with:

        --deqp-log-filename=<path>

By default, the shader cache will be written into the path "shadercache.bin". If the
platform requires a different path, it can be specified with:

        --deqp-shadercache-filename=<path>

If the shader cache is not desired, it can be disabled with:

        --deqp-shadercache=disable

No other command line options are allowed.

### Win32

        cd <builddir>/external/vulkancts/modules/vulkan
        Debug/deqp-vk.exe --deqp-caselist-file=...

Test log will be written into TestResults.qpa

### Linux

        cd <builddir>/external/vulkancts/modules/vulkan
        ./deqp-vk --deqp-vk-caselist-file=...

### Android

        adb push <vulkancts>/external/vulkancts/mustpass/1.1.4/vk-default.txt /sdcard/vk-default.txt
        adb shell

In device shell:

        am start -n com.drawelements.deqp/android.app.NativeActivity -e cmdLine "deqp --deqp-caselist-file=/sdcard/vk-default.txt --deqp-log-images=disable --deqp-log-shader-sources=disable --deqp-log-filename=/sdcard/TestResults.qpa"

Test progress will be written to device log and can be displayed with:

        adb logcat -s dEQP

Test log will be written into `/sdcard/TestResults.qpa`.


Conformance Submission Package Requirements
-------------------------------------------

The conformance submission package must contain the following:

1. Full test logs (`TestResults.qpa`) from CTS runs against all driver builds
2. Result of `git status` and `git log` from CTS source directory
3. Any patches used on top of release tag
4. Conformance statement

Test logs (1) should be named `<submission pkg dir>/TestResults-<driver build type>.qpa`,
for example `TestResults-armeabi-v7a.qpa`. On platforms where multiple different driver
builds (for example 64-bit and 32-bit) are present, CTS logs must be provided
for each driver build as part of the submission package.

Test logs generated on a system which exposes more than one physical device
in a device group can be used for products that expose one or more physical
devices in their device group.

The CTS build must always be done from clean git repository that doesn't have any
uncommitted changes. Thus it is necessary to run and capture output of `git
status` and `git log` (2) in the source directory:

        git status > <submission pkg dir>/git-status.txt
        git log --first-parent <release tag>^..HEAD > <submission pkg dir>/git-log.txt

Any changes made to CTS must be committed to the local repository, and provided
as part of the submission package (3). This can be done by running:

        git format-patch -o <submission pkg dir> <release tag>..HEAD

In general, bugfixes and changes to platform-specific code (mostly under
`framework/platform`) are allowed. The commit message for each change must
include a clear description of the change and why it is necessary. Non-porting
related changes must be accompanied by a waiver (see below).

NOTE: When cherry-picking patches on top of release tag, please use `git cherry-pick -x`
to include original commit hash in the commit message.

Conformance statement (4) must be included in a file called `STATEMENT-<adopter>`
and must contain following:

        CONFORM_VERSION:         <git tag of CTS release>
        PRODUCT:                 <string-value>
        CPU:                     <string-value>
        OS:                      <string-value>

Note that product/cpu/os information is also captured in `dEQP-VK.info.*` tests
if `vk::Platform::describePlatform()` is implemented.

If the submission package covers multiple products, you can list them by appending
additional `PRODUCT:` lines to the conformance statement. For example:

        CONFORM_VERSION:         vulkan-cts-1.1.4.0
        PRODUCT:                 Product A
        PRODUCT:                 Product B
        ...

The actual submission package consists of the above set of files which must
be bundled into a gzipped tar file named `VK<API major><API minor>_<adopter><_info>.tgz`.
`<API major>` is the major version of the Vulkan API specification, `<API minor>`is the minor
version of the Vulkan API specification.
`<adopter>` is the name of the Adopting member company, or some recognizable abbreviation.
The `<_info>` field is optional. It may be used to uniquely identify a submission
by OS, platform, date, or other criteria when making multiple submissions.
For example, a company XYZ may make a submission for a Vulkan 1.1 implementation named
`VK11_XYZ_PRODUCTA_Windows10.tgz`

One way to create a suiteable gzipped tar file is to execute the command:

        tar -cvzf <filename.tgz> -C <submission pkg dir> .

where `<submission pkg dir>` is the directory containing the files from (1)-(4)
from above. A submission package must contain all of the files listed above,
and only those files.

As an example submission package could contain:

        STATEMENT-Khronos
        git-log.txt
        git-status.txt
        0001-Remove-Waived-Filtering-Tests.patch
        0002-Fix-Pipeline-Parameters.patch
        TestResults-armeabi-v7a.qpa
        TestResults-arm64-v8a.qpa


Waivers
-------

The process for requesting a waiver is to report the issue by filing a bug
report in the Gitlab VulkanCTS project (TODO Github?). When creating the
submission package, include references to the waiver in the commit message of
the relevant change. Including as much information as possible in your bug
report (including a unified diff or a merge request of suggested file changes)
will ensure the issue can be progressed as rapidly as possible. Issues must
be labeled "Waiver" (TODO!) and identify the version of the CTS and affected
tests.

Conformance Criteria
--------------------

Conformance run is considered passing if all tests finish with allowed result
codes. Test results are contained in the TestResults.qpa log. Each
test case section contains XML tag Result, for example:

        <Result StatusCode="Pass">Not validated</Result>

The result code is the value of the StatusCode attribute. Following status
codes are allowed:

        Pass
        NotSupported
        QualityWarning
        CompatibilityWarning

Submission package can be verified using `external/vulkancts/scripts/verify_submission.py`
script. The script takes two arguments: path to extracted submission package
and path to current mustpass list. For example:

        python external/vulkancts/scripts/verify_submission.py VK_11_Khronos_1/ external/vulkancts/mustpass/1.1.4/vk-default.txt

Please note that the script reports a warning even for a correctly generated git-log.txt
If your git-log.txt contains only head commit of the release tag then
the warning can be ignored.

Vulkan platform port
--------------------

Vulkan support from Platform implementation requires providing
`getVulkanPlatform()` method in `tcu::Platform` class implementation.

See `framework/common/tcuPlatform.hpp` and examples in
`framework/platform/win32/tcuWin32Platform.cpp` and
`framework/platform/android/tcuAndroidPlatform.cpp`.

If any WSI extensions are supported, platform port must also implement
methods for creating native display (`vk::Platform::createWsiDisplay`)
and window handles (`vk::wsi::Display::createWindow`). Otherwise tests
under `dEQP-VK.wsi` will fail.


Null (dummy) driver
-------------------

For testing and development purposes it might be useful to be able to run
tests on dummy Vulkan implementation. One such implementation is provided in
vkNullDriver.cpp. To use that, implement `vk::Platform::createLibrary()` with
`vk::createNullDriver()`.


Validation Layers
-----------------

Vulkan CTS framework includes first-party support for validation layers, that
can be turned on with `--deqp-validation=enable` command line option.

When validation is turned on, default instance and device will be created with
validation layers enabled and debug callback is registered to record any
messages. Debug messages collected during test execution will be included at
the end of the test case log.

If any validation errors are found, test result will be set to `InternalError`.

By default `VK_DEBUG_REPORT_INFORMATION_BIT_EXT` and `_DEBUG_BIT_EXT` messages
are excluded from the log, but that can be customized by modifying
`vkt::TestCaseExecutor::deinit()` in `vktTestPackage.cpp`.

On the Android target, layers can be added to the APK during the build process
by setting the `--layers-path` command line option to point into the NDK or to
a locally-built layers tree. The layers are expected to be found under $abi/
under the layers path.


Cherry GUI
----------

Vulkan test module can be used with Cherry (GUI for test execution and
analysis). Cherry is available at
https://android.googlesource.com/platform/external/cherry. Please follow
instructions in README to get started.

Before first launch, and every time test hierarchy has been modified, test
case list must be refreshed by running:

        python scripts/build_caselists.py path/to/cherry/data

Cherry must be restarted for the case list update to take effect.


Shader Optimizer
----------------

Vulkan CTS can be optionally run with the shader optimizer enabled. This
is an experimental feature which can be used to further stress both the
drivers as well as the optimizer itself. The shader optimizer is disabled
by default.

The following command line options can be used to configure the shader
optimizer:

        --deqp-optimization-recipe=<number>

The list of the optimization recipes can be found and customized in the
`optimizeCompiledBinary()` function in `vkPrograms.cpp`.

As of this writing, there are 8 recipes to choose from:

        0. Disabled (default)
        1. The example recipe from spir-v opt 1.0 whitepaper
        2. RegisterPerformancePasses from commandline optimizer tool october 2017
        3. RegisterSizePasses from commandline optimizer tool october 2017
        4. RegisterLegalizationPasses from commandline optimizer tool April 2018
        5. RegisterPerformancePasses from commandline optimizer tool April 2018
        6. RegisterPerformancePasses from commandline optimizer tool April 2018 with CreateCommonUniformElimPass
        7. RegisterSizePasses from commandline optimizer tool April 2018
        8. RegisterSizePasses from commandline optimizer tool April 2018 with CreateCommonUniformElimPass

The `Register...Passes()` calls change in the SPIR-V optimizer tool from
time to time. Since different sets of passes may result in different
shaders, having several fixed sets is useful for issue discovery.

        --deqp-optimize-spirv=enable

This option is not required to run the optimizer. By default, the shader
optimizer only optimizes shaders generated from GLSL or HLSL, and leaves
hand-written SPIR-V shaders alone.

Many of the hand-written SPIR-V tests stress specific features of the
SPIR-V which might get optimized out. Using this option will enable the
optimizer on the hand-written SPIR-V as well, which may be useful in
finding new bugs in drivers or the optimizer itself, but will likely
invalidate the tests themselves.


Shader Cache
------------

The Vulkan CTS framework contains a shader cache for speeding up the running
of the CTS. Skipping shader compilation can significantly reduce runtime,
especially for repeated runs.

Default behavior is to have the shader cache enabled, but truncated at the
start of the CTS run. This still gives the benefit of skipping shader
compilation for identical shaders in different tests (which there are many),
while making sure that the shader cache file does not grow indefinitely.

The shader cache identifies the shaders by hashing the shader source code
along with various bits of information that may affect the shader compilation
(such as shader stage, CTS version, possible compilation flags, etc). If a
cached shader with matching hash is found, a byte-by-byte comparison of the
shader sources is made to make sure that the correct shader is being
retrieved from the cache.

The behavior of the shader cache can be modified with the following command
line options:

        --deqp-shadercache=disable

Disable the shader cache. All shaders will be compiled every time.

        --deqp-shadercache-filename=<filename>

Set the name of the file where the cached shaders will be stored. This
option may be required for the shader cache to work at all on Android
targets.

        --deqp-shadercache-truncate=disable

Do not truncate the shader cache file at startup. No shader compilation will
occur on repeated runs of the CTS.


RenderDoc
---------
The RenderDoc (https://renderdoc.org/) graphics debugger may be used to debug
Vulkan tests.

Following command line option should be used when launching tests from
RenderDoc UI:

        --deqp-renderdoc=enable

This causes the framework to interface with the debugger and mark each dEQP
test case as a separate 'frame', just for the purpose of capturing. The frames
are added using RenderDoc 'In-Application API', instead of swapchain operations.