[nnc] Add readme for interpreter testing system (#1157)

Described how to run interpreter tests and how to add
tests for new NN operations.

Signed-off-by: Dmitry Mozolev <d.mozolev@samsung.com>

diff --git a/contrib/nnc/doc/nnc_interpreter_testing.rst b/contrib/nnc/doc/nnc_interpreter_testing.rst
new file mode 100644 (file)
index 0000000..af733e2
--- /dev/null
+++ b/contrib/nnc/doc/nnc_interpreter_testing.rst
@@ -0,0 +1,114 @@
+NNC Interpreter Testing System
+##############################
+
+Requirements
+************
+
+To generate testing data you need Python 3 and following Python libraries:
+
+#. tensorflow
+#. flatbuffers
+#. numpy
+
+All of them are listed in `gen/requirements.txt` and can be installed using `pip -r requirements.txt`.
+
+Step 1: Generating test data
+****************************
+
+#. `cd gen`
+#. `./run_gen.sh`
+
+After this a number of `.fb` files will appear in `test_data` directory. Each of them is a flatbuffers-serialized file containing test data - an array of structures containing information about a single NN operation.
+
+This information includes:
+
+* Type
+* Inputs
+* Outputs
+* Kernels (if applicable)
+* Padding type (if applicable)
+* Strides (if applicable)
+* Other parameters like pooling type, axis etc (if applicable)
+
+Step 2: Running tests
+*********************
+
+Running cmake after generating the test data will produce a number of test targets named "interpreter_<op_name>_test". They can be run using:
+
+`ctest -R "interp"`
+
+from the cmake build directory.
+
+Adding new operation type to the testing system
+***********************************************
+
+Flatbuffers schema describing serialized test info file layout is located in the `op_info.fbs` file.
+
+First, new operation name should be added to the `enum OperatorType`.
+
+Second, information about the parameters that this operation takes should be added to the `OP_FORMATS` dict in the `gen/gen_test_data.py` script.
+
+This information is represented by a tuple of strings, which denote the order of operation parameters in the `test_data/test_description.txt` file (see `Test description file format <test_descr_>`_). At the moment, each string is one of:
+
+* `kernels`
+* `padType`
+* `poolType`
+* `shapes`
+* `axis`
+
+Third, a section describing test parameters for the new operation should be added to `test_data/test_description.txt` file, analagously to other operations in there (see `Test description file format <test_descr_>`_).
+
+Fourth, another method for generating the operation result should be added to the `OpInfoProvider` class in `gen/get_test_data.py` script. This method should use Tensorflow to calculate the reference result for a given set of operation parameters, analagously to how it is done for the other operation types. Note that the name of the method **has** to be of the form `get_<lowercase_op_name_as_in_test_description.txt>_result`.
+
+.. _test_descr:
+
+Test description file format
+****************************
+
+This file consists of lines, where each line is one of:
+
+* an empty line
+* a comment, starting with `#`
+* an operation name
+* operation parameters description
+
+A line with an operation name acts a start of a block of lines containing parameter descriptions for this operation.
+
+Each line with an operation parameter description should follow the pattern described in the `OP_FORMATS` dict in the `gen/get_test_data.py` script. Each operation has `inputs` implicitly as a first element, and `inputs` description is always first in every operation parameter description line.
+
+Example of operation parameters description line for CONV_2D operation:
+
+`[64, 64, 4] [3, 3, 4, 2] VALID [1, 1]`
+
+* `[64, 64, 4]` means that for this operation an input of shape `[64, 64, 4]` will be generated
+* `[3, 3, 4, 2]` means that for this operation a kernel of shape `[3, 3, 4, 2]` will be generated
+* `VALID` denotes padding type
+* `[1, 1]` denotes strides
+
+This format follows what can be found in the `OP_FORMATS` map: `'CONV_2D': ('kernels', 'padType', 'shapes')`.
+
+In general, there may be multiple inputs, kernels or shapes that operation has. In this case, multiple parameters of the same type, like `shapes` for example, should be wrapped in an additional pair of square brackets.
+
+For example for `POOL_2D`:
+
+`[64, 64, 4] VALID MAX [[3, 3] [1, 1]]`
+
+* `[3, 3]` denotes pooling window shape
+* `[1, 1]` denotes strides
+
+As both windows shape and strides are parameters of type `shapes` as denoted in `OP_FORMATS` dict - `'POOL_2D': ('padType', 'poolType', 'shapes')` - they are grouped using another pair of square brackets (note the absence of commas in there).
+
+The fact that the first shape is interpreted as a pooling window shape, and the second as strides, is represented in the C++ part of this testing system, where these serialized files are read and Model IR graph is constructed.
+
+Random tensor generation
+************************
+
+Currently, random values for the inputs and kernels are taken from a uniform distribution over [-5, 5).
+
+This is not the only way to randomly initialize the tensors, and other methods might also be desirable in the future. One way to support this could be to define a number of named (and possibly parametrized) initialization methods, and add them to input and kernel shape descriptions, for example like this:
+
+.. code-block::
+
+    CONV_2D
+    [28, 28, 3]=UNIFORM(-2, 3) [3, 3, 3, 10]=NEAR_ZERO VALID [1, 1]
+