From: Роман Михайлович Русяев/AI Tools Lab /SRR/Staff Engineer/삼성전자 Date: Wed, 28 Nov 2018 14:49:10 +0000 (+0300) Subject: Add Test Round 2.1 in System Test Document (#2329) X-Git-Tag: nncc_backup~1242 X-Git-Url: http://review.tizen.org/git/?a=commitdiff_plain;h=d2d5b1f971fb94fa8641df8e9a2ca941b603f9e1;p=platform%2Fcore%2Fml%2Fnnfw.git Add Test Round 2.1 in System Test Document (#2329) * added tests for Tizen and SmartMachine OS Signed-off-by: Roman Rusyaev --- diff --git a/contrib/nnc/doc/project/18_NN_Compiler_and_Optimizer_STD.rst b/contrib/nnc/doc/project/18_NN_Compiler_and_Optimizer_STD.rst index 11bb324..e4f5d48 100644 --- a/contrib/nnc/doc/project/18_NN_Compiler_and_Optimizer_STD.rst +++ b/contrib/nnc/doc/project/18_NN_Compiler_and_Optimizer_STD.rst @@ -27,6 +27,8 @@ SW System Test Document +-------+-------------+----------------------------+--------------------------+---------------------+ | 2.1 | 2018.10.23 | Test Round 2 | Roman Rusyaev | Sung-Jae Lee | +-------+-------------+----------------------------+--------------------------+---------------------+ +| 2.2 | 2018.11.09 | Test Round 2.1 | Roman Rusyaev | Sung-Jae Lee | ++-------+-------------+----------------------------+--------------------------+---------------------+ | @@ -152,7 +154,7 @@ Functions to be tested * - RF-7, RF-18, RIF-13 - TST-7 - - The unit test should automatically verify successful execution of binary on target ARM CPU + - The test should automatically verify successful execution of binary on ARM CPU * - RF-8, RF-17, RIF-14, RIF-15 - TST-8 @@ -190,6 +192,7 @@ Functions to be tested - TST-16 - The unit test should verify the existence and validity of generated C/C++ header for compiled artefact + **The following requirements can be tested only manually:** * Non-functional requirements: RNF-2, RNF-3 (They would be tested during development) @@ -527,6 +530,11 @@ Baseline - SRR-RAJ0118ZZ-BWRF-STD002, SRR-RAJ0118ZZ-BWRF-UTR002 - 11.2018 + * - Round 2.1 + - The nncc v0.9 (Commit: **2d7efd2d0bc8a1ba135cb4996902284078e14eff**) + - SRR-RAJ0118ZZ-BWRF-STD002, SRR-RAJ0118ZZ-BWRF-UTR002 + - 11.2018 + * - Round 3 - The nncc v1.0 - SRR-RAJ0118ZZ-BWRF-STD003, SRR-RAJ0118ZZ-BWRF-UTR003 @@ -1260,12 +1268,32 @@ The following tests are not supported because they are unavailable in ML Framewo * - TST-7 - TST-7-1 - - Generate binary for the Inception V3 NN model and run its inference on a device with ARM CPU + - Generate binary for the Inception V3 NN model and run its inference on ARM CPU with Ubuntu - Test executed successfully, no crashes occurred, inference result was output, amount and format of the outputs corresponds to the expected NN model outputs * - TST-7 - TST-7-2 - - Generate binary for the MobileNet NN model and run its inference on a device with ARM CPU + - Generate binary for the MobileNet NN model and run its inference on ARM CPU with Ubuntu + - Test executed successfully, no crashes occurred, inference result was output, amount and format of the outputs corresponds to the expected NN model outputs + + * - TST-7 + - TST-7-3 + - Generate binary for the Inception V3 NN model and run its inference on ARM CPU with Tizen + - Test executed successfully, no crashes occurred, inference result was output, amount and format of the outputs corresponds to the expected NN model outputs + + * - TST-7 + - TST-7-4 + - Generate binary for the MobileNet NN model and run its inference on ARM CPU with Tizen + - Test executed successfully, no crashes occurred, inference result was output, amount and format of the outputs corresponds to the expected NN model outputs + + * - TST-7 + - TST-7-5 + - Generate binary for the Inception V3 NN model and run its inference on ARM CPU with SmartMachine OS + - Test executed successfully, no crashes occurred, inference result was output, amount and format of the outputs corresponds to the expected NN model outputs + + * - TST-7 + - TST-7-6 + - Generate binary for the MobileNet NN model and run its inference on ARM CPU with SmartMachine OS - Test executed successfully, no crashes occurred, inference result was output, amount and format of the outputs corresponds to the expected NN model outputs * - TST-8 @@ -1446,6 +1474,8 @@ Test Schedule +------------+-----------+---------+-----------+--------+-----------------+ | R.2 | 11/2018 | 1 | 10/2018 | 1 | | +------------+-----------+---------+-----------+--------+-----------------+ +| R.2.1 | 11/2018 | 1 | 11/2018 | 1 | | ++------------+-----------+---------+-----------+--------+-----------------+ | R.3 | 12/2018 | 1 | | | | +------------+-----------+---------+-----------+--------+-----------------+ @@ -1497,6 +1527,27 @@ Test Round 2 Expected failures: 0 Unexpected successes: 0 +Test Round 2.1 +`````````````` + +:: + + Discovering tests... + Total tests discovered: 122 + Log folder: /home/ivan/GIT/nnc_doc_test/2018_11_09_21_46_03 + Threads quantity: 1 + + ================================= + Tests: 122/122, successes: 122, failures: 0, errors: 0, other: 0 + ================================= + Result: + Total tests: 122 + Successes: 122 + Errors: 0 + Failures: 0 + Skipped: 0 + Expected failures: 0 + Unexpected successes: 0 Test Result ----------- @@ -1506,13 +1557,15 @@ Test Result +--------------------+------------------------------------+------------+----------------+ | Official release | The number of test case | Pass rate | Result | -+ +-------+-------+------+-------------+ | | +| +-------+-------+------+-------------+ | | | | Total | Pass | Fail | Unsupported | Pass/Total | | +====================+=======+=======+======+=============+============+================+ | v0.3 | 80 | 80 | 0 | 1 | 1.0 |**Tests passed**| +--------------------+-------+-------+------+-------------+------------+----------------+ | v0.7 | 118 | 118 | 0 | 15 | 1.0 |**Tests passed**| +--------------------+-------+-------+------+-------------+------------+----------------+ +| v0.9 | 122 | 122 | 0 | 15 | 1.0 |**Tests passed**| ++--------------------+-------+-------+------+-------------+------------+----------------+ | @@ -1527,7 +1580,55 @@ Test Result +---------+----------+--------+-------+----------+--------+-------+ | R.2 | 0 | 0 | 0 | 0 | 0 | 0 | +---------+----------+--------+-------+----------+--------+-------+ +| R.2.1 | 0 | 0 | 0 | 0 | 0 | 0 | ++---------+----------+--------+-------+----------+--------+-------+ + +| +**Table 6-5. Round Test Item** + ++---------+----------------------------------------------------------------------------------------------+ +| Round | Test Item | ++=========+==============================================================================================+ +| R.1 | TST-2-1, TST-2-2, TST-2-3, TST-2-4, TST-2-5, TST-2-6, TST-2-7, TST-2-8, TST-2-9, | +| | TST-2-10, TST-2-11, TST-2-12, TST-2-13, TST-2-14, TST-2-16, TST-2-17, TST-2-18, TST-2-19, | +| | TST-2-20, TST-2-21, TST-2-22, TST-2-23, TST-2-24, TST-2-25, TST-2-26, TST-2-27, TST-2-28, | +| | TST-2-29, TST-2-30, TST-2-31, TST-2-32, TST-2-33, TST-2-34, TST-2-35, TST-2-36, TST-2-37, | +| | TST-2-38, TST-2-39, TST-2-40, TST-2-41, TST-2-42, TST-2-43, TST-2-44, TST-2-45, TST-2-46, | +| | TST-2-47, TST-2-48, TST-2-49, TST-2-50, TST-2-51, TST-2-52, TST-2-53, TST-2-54, TST-2-55, | +| | TST-2-56, TST-2-57, TST-2-58, TST-4-1, TST-5-1, TST-7-1, TST-7-2, TST-9-1, TST-9-2, TST-9-3, | +| | TST-9-4, TST-10-1, TST-10-2, TST-11-1, TST-11-2, TST-13-1, TST-13-2, TST-13-5, TST-13-6, | +| | TST-14-1, TST-14-2, TST-14-3, TST-14-4, TST-15-1, TST-16-1, TST-16-2 | ++---------+----------------------------------------------------------------------------------------------+ +| R.2 | TST-2-1, TST-2-2, TST-2-3, TST-2-4, TST-2-5, TST-2-6, TST-2-7, TST-2-8, TST-2-9, | +| | TST-2-10, TST-2-11, TST-2-12, TST-2-13, TST-2-14, TST-2-16, TST-2-17, TST-2-18, TST-2-19, | +| | TST-2-20, TST-2-21, TST-2-22, TST-2-23, TST-2-24, TST-2-25, TST-2-26, TST-2-27, TST-2-28, | +| | TST-2-29, TST-2-30, TST-2-31, TST-2-32, TST-2-33, TST-2-34, TST-2-35, TST-2-36, TST-2-37, | +| | TST-2-38, TST-2-39, TST-2-40, TST-2-41, TST-2-42, TST-2-43, TST-2-44, TST-2-45, TST-2-46, | +| | TST-2-47, TST-2-48, TST-2-49, TST-2-50, TST-2-51, TST-2-52, TST-2-53, TST-2-54, TST-2-55, | +| | TST-2-56, TST-2-57, TST-2-58, TST-4-1, TST-5-1, TST-7-1, TST-7-2, TST-9-1, TST-9-2, TST-9-3, | +| | TST-9-4, TST-10-1, TST-10-2, TST-11-1, TST-11-2, TST-13-1, TST-13-2, TST-13-5, TST-13-6, | +| | TST-14-1, TST-14-2, TST-14-3, TST-14-4, TST-15-1, TST-16-1, TST-16-2, TST-1-11, TST-1-25, | +| | TST-1-36, TST-1-37, TST-1-10, TST-1-12, TST-1-13, TST-1-16, TST-1-17, TST-1-18, TST-1-19, | +| | TST-1-1, TST-1-20, TST-1-21, TST-1-22, TST-1-24, TST-1-2, TST-1-33, TST-1-35, TST-1-38, | +| | TST-1-39, TST-1-3, TST-1-40, TST-1-41, TST-1-42, TST-1-44, TST-1-46, TST-1-47, TST-1-48, | +| | TST-1-49, TST-1-4, TST-1-50, TST-1-52, TST-1-5, TST-1-6, TST-1-7, TST-1-8, TST-12-1 | ++---------+----------------------------------------------------------------------------------------------+ +| R.2.1 | TST-2-1, TST-2-2, TST-2-3, TST-2-4, TST-2-5, TST-2-6, TST-2-7, TST-2-8, TST-2-9, | +| | TST-2-10, TST-2-11, TST-2-12, TST-2-13, TST-2-14, TST-2-16, TST-2-17, TST-2-18, TST-2-19, | +| | TST-2-20, TST-2-21, TST-2-22, TST-2-23, TST-2-24, TST-2-25, TST-2-26, TST-2-27, TST-2-28, | +| | TST-2-29, TST-2-30, TST-2-31, TST-2-32, TST-2-33, TST-2-34, TST-2-35, TST-2-36, TST-2-37, | +| | TST-2-38, TST-2-39, TST-2-40, TST-2-41, TST-2-42, TST-2-43, TST-2-44, TST-2-45, TST-2-46, | +| | TST-2-47, TST-2-48, TST-2-49, TST-2-50, TST-2-51, TST-2-52, TST-2-53, TST-2-54, TST-2-55, | +| | TST-2-56, TST-2-57, TST-2-58, TST-4-1, TST-5-1, TST-7-1, TST-7-2, TST-7-3, TST-7-4, TST-7-5, | +| | TST-7-6, TST-9-1, TST-9-2, TST-9-3, TST-9-4, TST-10-1, TST-10-2, TST-11-1, TST-11-2, | +| | TST-13-1, TST-13-2, TST-13-5, TST-13-6, TST-14-1, TST-14-2, TST-14-3, TST-14-4, TST-15-1, | +| | TST-16-1, TST-16-2, TST-1-11, TST-1-25, TST-1-36, TST-1-37, TST-1-10, TST-1-12, TST-1-13, | +| | TST-1-16, TST-1-17, TST-1-18, TST-1-19, TST-1-1, TST-1-20, TST-1-21, TST-1-22, TST-1-24, | +| | TST-1-2, TST-1-33, TST-1-35, TST-1-38, TST-1-39, TST-1-3, TST-1-40, TST-1-41, TST-1-42, | +| | TST-1-44, TST-1-46, TST-1-47, TST-1-48, TST-1-49, TST-1-4, TST-1-50, TST-1-52, TST-1-5, | +| | TST-1-6, TST-1-7, TST-1-8, TST-12-1 | ++---------+----------------------------------------------------------------------------------------------+ Defect rates ````````````