Add Test Round 2.1 in System Test Document (#2329)

* added tests for Tizen and SmartMachine OS

Signed-off-by: Roman Rusyaev <r.rusyaev@samsung.com>

diff --git a/contrib/nnc/doc/project/18_NN_Compiler_and_Optimizer_STD.rst b/contrib/nnc/doc/project/18_NN_Compiler_and_Optimizer_STD.rst
index 11bb324..e4f5d48 100644 (file)
--- a/contrib/nnc/doc/project/18_NN_Compiler_and_Optimizer_STD.rst
+++ b/contrib/nnc/doc/project/18_NN_Compiler_and_Optimizer_STD.rst
@@ -27,6 +27,8 @@ SW System Test Document
 +-------+-------------+----------------------------+--------------------------+---------------------+
 | 2.1   | 2018.10.23  | Test Round 2               | Roman Rusyaev            | Sung-Jae Lee        |
 +-------+-------------+----------------------------+--------------------------+---------------------+
+| 2.2   | 2018.11.09  | Test Round 2.1             | Roman Rusyaev            | Sung-Jae Lee        |
++-------+-------------+----------------------------+--------------------------+---------------------+
 
 |
 
@@ -152,7 +154,7 @@ Functions to be tested
 
    * - RF-7, RF-18, RIF-13
      - TST-7
-     - The unit test should automatically verify successful execution of binary on target ARM CPU
+     - The test should automatically verify successful execution of binary on ARM CPU
 
    * - RF-8, RF-17, RIF-14, RIF-15
      - TST-8
@@ -190,6 +192,7 @@ Functions to be tested
      - TST-16
      - The unit test should verify the existence and validity of generated C/C++ header for compiled artefact
 
+
 **The following requirements can be tested only manually:**
 
 * Non-functional requirements: RNF-2, RNF-3 (They would be tested during development)
@@ -527,6 +530,11 @@ Baseline
      - SRR-RAJ0118ZZ-BWRF-STD002, SRR-RAJ0118ZZ-BWRF-UTR002
      - 11.2018
 
+   * - Round 2.1
+     - The nncc v0.9 (Commit: **2d7efd2d0bc8a1ba135cb4996902284078e14eff**)
+     - SRR-RAJ0118ZZ-BWRF-STD002, SRR-RAJ0118ZZ-BWRF-UTR002
+     - 11.2018
+
    * - Round 3
      - The nncc v1.0
      - SRR-RAJ0118ZZ-BWRF-STD003, SRR-RAJ0118ZZ-BWRF-UTR003
@@ -1260,12 +1268,32 @@ The following tests are not supported because they are unavailable in ML Framewo
 
    * - TST-7
      - TST-7-1
-     - Generate binary for the Inception V3 NN model and run its inference on a device with ARM CPU
+     - Generate binary for the Inception V3 NN model and run its inference on ARM CPU with Ubuntu
      - Test executed successfully, no crashes occurred, inference result was output, amount and format of the outputs corresponds to the expected NN model outputs
 
    * - TST-7
      - TST-7-2
-     - Generate binary for the MobileNet NN model and run its inference on a device with ARM CPU
+     - Generate binary for the MobileNet NN model and run its inference on ARM CPU with Ubuntu
+     - Test executed successfully, no crashes occurred, inference result was output, amount and format of the outputs corresponds to the expected NN model outputs
+
+   * - TST-7
+     - TST-7-3
+     - Generate binary for the Inception V3 NN model and run its inference on ARM CPU with Tizen
+     - Test executed successfully, no crashes occurred, inference result was output, amount and format of the outputs corresponds to the expected NN model outputs
+
+   * - TST-7
+     - TST-7-4
+     - Generate binary for the MobileNet NN model and run its inference on ARM CPU with Tizen
+     - Test executed successfully, no crashes occurred, inference result was output, amount and format of the outputs corresponds to the expected NN model outputs
+
+   * - TST-7
+     - TST-7-5
+     - Generate binary for the Inception V3 NN model and run its inference on ARM CPU with SmartMachine OS
+     - Test executed successfully, no crashes occurred, inference result was output, amount and format of the outputs corresponds to the expected NN model outputs
+
+   * - TST-7
+     - TST-7-6
+     - Generate binary for the MobileNet NN model and run its inference on ARM CPU with SmartMachine OS
      - Test executed successfully, no crashes occurred, inference result was output, amount and format of the outputs corresponds to the expected NN model outputs
 
    * - TST-8
@@ -1446,6 +1474,8 @@ Test Schedule
 +------------+-----------+---------+-----------+--------+-----------------+
 | R.2        |  11/2018  |   1     |  10/2018  |  1     |                 |
 +------------+-----------+---------+-----------+--------+-----------------+
+| R.2.1      |  11/2018  |   1     |  11/2018  |  1     |                 |
++------------+-----------+---------+-----------+--------+-----------------+
 | R.3        |  12/2018  |   1     |           |        |                 |
 +------------+-----------+---------+-----------+--------+-----------------+
 
@@ -1497,6 +1527,27 @@ Test Round 2
  Expected failures:    0
  Unexpected successes: 0
 
+Test Round 2.1
+``````````````
+
+::
+
+ Discovering tests...
+ Total tests discovered: 122
+ Log folder:             /home/ivan/GIT/nnc_doc_test/2018_11_09_21_46_03
+ Threads quantity:       1
+
+ =================================
+ Tests: 122/122, successes: 122, failures: 0, errors: 0, other: 0
+ =================================
+ Result:
+ Total tests:          122
+ Successes:            122
+ Errors:               0
+ Failures:             0
+ Skipped:              0
+ Expected failures:    0
+ Unexpected successes: 0
 
 Test Result
 -----------
@@ -1506,13 +1557,15 @@ Test Result
 
 +--------------------+------------------------------------+------------+----------------+
 | Official release   |      The number of test case       | Pass rate  |    Result      |
-+                    +-------+-------+------+-------------+            |                |
+|                    +-------+-------+------+-------------+            |                |
 |                    | Total | Pass  | Fail | Unsupported | Pass/Total |                |
 +====================+=======+=======+======+=============+============+================+
 |  v0.3              |  80   | 80    |  0   |      1      |   1.0      |**Tests passed**|
 +--------------------+-------+-------+------+-------------+------------+----------------+
 |  v0.7              |  118  | 118   |  0   |      15     |   1.0      |**Tests passed**|
 +--------------------+-------+-------+------+-------------+------------+----------------+
+|  v0.9              |  122  | 122   |  0   |      15     |   1.0      |**Tests passed**|
++--------------------+-------+-------+------+-------------+------------+----------------+
 
 |
 
@@ -1527,7 +1580,55 @@ Test Result
 +---------+----------+--------+-------+----------+--------+-------+
 |   R.2   |    0     |  0     |   0   |   0      |   0    |  0    |
 +---------+----------+--------+-------+----------+--------+-------+
+|   R.2.1 |    0     |  0     |   0   |   0      |   0    |  0    |
++---------+----------+--------+-------+----------+--------+-------+
+
+|
 
+**Table 6-5. Round Test Item**
+
++---------+----------------------------------------------------------------------------------------------+
+| Round   |                                   Test Item                                                  |
++=========+==============================================================================================+
+|   R.1   | TST-2-1, TST-2-2, TST-2-3, TST-2-4, TST-2-5, TST-2-6, TST-2-7, TST-2-8, TST-2-9,             |
+|         | TST-2-10, TST-2-11, TST-2-12, TST-2-13, TST-2-14, TST-2-16, TST-2-17, TST-2-18, TST-2-19,    |
+|         | TST-2-20, TST-2-21, TST-2-22, TST-2-23, TST-2-24, TST-2-25, TST-2-26, TST-2-27, TST-2-28,    |
+|         | TST-2-29, TST-2-30, TST-2-31, TST-2-32, TST-2-33, TST-2-34, TST-2-35, TST-2-36, TST-2-37,    |
+|         | TST-2-38, TST-2-39, TST-2-40, TST-2-41, TST-2-42, TST-2-43, TST-2-44, TST-2-45, TST-2-46,    |
+|         | TST-2-47, TST-2-48, TST-2-49, TST-2-50, TST-2-51, TST-2-52, TST-2-53, TST-2-54, TST-2-55,    |
+|         | TST-2-56, TST-2-57, TST-2-58, TST-4-1, TST-5-1, TST-7-1, TST-7-2, TST-9-1, TST-9-2, TST-9-3, |
+|         | TST-9-4, TST-10-1, TST-10-2, TST-11-1, TST-11-2, TST-13-1, TST-13-2, TST-13-5, TST-13-6,     |
+|         | TST-14-1, TST-14-2, TST-14-3, TST-14-4, TST-15-1, TST-16-1, TST-16-2                         |
++---------+----------------------------------------------------------------------------------------------+
+|   R.2   | TST-2-1, TST-2-2, TST-2-3, TST-2-4, TST-2-5, TST-2-6, TST-2-7, TST-2-8, TST-2-9,             |
+|         | TST-2-10, TST-2-11, TST-2-12, TST-2-13, TST-2-14, TST-2-16, TST-2-17, TST-2-18, TST-2-19,    |
+|         | TST-2-20, TST-2-21, TST-2-22, TST-2-23, TST-2-24, TST-2-25, TST-2-26, TST-2-27, TST-2-28,    |
+|         | TST-2-29, TST-2-30, TST-2-31, TST-2-32, TST-2-33, TST-2-34, TST-2-35, TST-2-36, TST-2-37,    |
+|         | TST-2-38, TST-2-39, TST-2-40, TST-2-41, TST-2-42, TST-2-43, TST-2-44, TST-2-45, TST-2-46,    |
+|         | TST-2-47, TST-2-48, TST-2-49, TST-2-50, TST-2-51, TST-2-52, TST-2-53, TST-2-54, TST-2-55,    |
+|         | TST-2-56, TST-2-57, TST-2-58, TST-4-1, TST-5-1, TST-7-1, TST-7-2, TST-9-1, TST-9-2, TST-9-3, |
+|         | TST-9-4, TST-10-1, TST-10-2, TST-11-1, TST-11-2, TST-13-1, TST-13-2, TST-13-5, TST-13-6,     |
+|         | TST-14-1, TST-14-2, TST-14-3, TST-14-4, TST-15-1, TST-16-1, TST-16-2, TST-1-11, TST-1-25,    |
+|         | TST-1-36, TST-1-37, TST-1-10, TST-1-12, TST-1-13, TST-1-16, TST-1-17, TST-1-18, TST-1-19,    |
+|         | TST-1-1, TST-1-20, TST-1-21, TST-1-22, TST-1-24, TST-1-2, TST-1-33, TST-1-35, TST-1-38,      |
+|         | TST-1-39, TST-1-3, TST-1-40, TST-1-41, TST-1-42, TST-1-44, TST-1-46, TST-1-47, TST-1-48,     |
+|         | TST-1-49, TST-1-4, TST-1-50, TST-1-52, TST-1-5, TST-1-6, TST-1-7, TST-1-8, TST-12-1          |
++---------+----------------------------------------------------------------------------------------------+
+|   R.2.1 | TST-2-1, TST-2-2, TST-2-3, TST-2-4, TST-2-5, TST-2-6, TST-2-7, TST-2-8, TST-2-9,             |
+|         | TST-2-10, TST-2-11, TST-2-12, TST-2-13, TST-2-14, TST-2-16, TST-2-17, TST-2-18, TST-2-19,    |
+|         | TST-2-20, TST-2-21, TST-2-22, TST-2-23, TST-2-24, TST-2-25, TST-2-26, TST-2-27, TST-2-28,    |
+|         | TST-2-29, TST-2-30, TST-2-31, TST-2-32, TST-2-33, TST-2-34, TST-2-35, TST-2-36, TST-2-37,    |
+|         | TST-2-38, TST-2-39, TST-2-40, TST-2-41, TST-2-42, TST-2-43, TST-2-44, TST-2-45, TST-2-46,    |
+|         | TST-2-47, TST-2-48, TST-2-49, TST-2-50, TST-2-51, TST-2-52, TST-2-53, TST-2-54, TST-2-55,    |
+|         | TST-2-56, TST-2-57, TST-2-58, TST-4-1, TST-5-1, TST-7-1, TST-7-2, TST-7-3, TST-7-4, TST-7-5, |
+|         | TST-7-6, TST-9-1, TST-9-2, TST-9-3, TST-9-4, TST-10-1, TST-10-2, TST-11-1, TST-11-2,         |
+|         | TST-13-1, TST-13-2, TST-13-5, TST-13-6, TST-14-1, TST-14-2, TST-14-3, TST-14-4, TST-15-1,    |
+|         | TST-16-1, TST-16-2, TST-1-11, TST-1-25, TST-1-36, TST-1-37, TST-1-10, TST-1-12, TST-1-13,    |
+|         | TST-1-16, TST-1-17, TST-1-18, TST-1-19,  TST-1-1, TST-1-20, TST-1-21, TST-1-22, TST-1-24,    |
+|         | TST-1-2, TST-1-33, TST-1-35, TST-1-38, TST-1-39, TST-1-3, TST-1-40, TST-1-41, TST-1-42,      |
+|         | TST-1-44, TST-1-46, TST-1-47, TST-1-48, TST-1-49, TST-1-4, TST-1-50, TST-1-52, TST-1-5,      |
+|         | TST-1-6, TST-1-7, TST-1-8, TST-12-1                                                          |
++---------+----------------------------------------------------------------------------------------------+
 
 Defect rates
 ````````````