From 03c4e9f03f6f9c9102b70e646d8fab42ce2138f0 Mon Sep 17 00:00:00 2001 From: =?utf8?q?Sergey=20Vostokov/SRR-AI=20Tools=20Lab/=2E/=EC=82=BC?= =?utf8?q?=EC=84=B1=EC=A0=84=EC=9E=90?= Date: Fri, 4 May 2018 09:00:02 +0300 Subject: [PATCH] SW System Test Document v1.0 (#77) * Add initital version of Test Plan Document This commit adds the initital version for SW Test Plan document Signed-off-by: Sergey Vostokov * Add section SW System Test Items This commit adds the list of functions to be tested and not to be tested Signed-off-by: Sergey Vostokov * Fix mistakes This commit fixes some mistakes in text Signed-off-by: Sergey Vostokov * Add software items to table 4-3 Add CI and VCS systems to table 4-3. Software Signed-off-by: Sergey Vostokov * Add risk management plan This commit adds the risk management plan to the section 4.3 Signed-off-by: Sergey Vostokov * Update risk management table The risk "SmartMachine OS targets are not available" was added Signed-off-by: Sergey Vostokov * Update formulation of the Scope section Some changes to reflect the exact scope. Signed-off-by: Sergey Vostokov * Change RST format to MD This commit adds the same content of Test Plan, but in MD format Signed-off-by: Sergey Vostokov * Update SW Test document by comments from SRR SE Team This commit add more details in Scope, updates test items, and test tools Signed-off-by: Sergey Vostokov * Fix format errors Fix format errors Signed-off-by: Sergey Vostokov * Remove test item 1 and test item 2. Update numbering This commit removes test items1/2 as they are considered as a part of other tests Signed-off-by: Sergey Vostokov * Update test tescription for test item 2/3 This commit updates the test items description for the test item 2 and test item 3 Signed-off-by: Sergey Vostokov * Update SW Test Document to follow SW Requirements Specification This commit updates test Item to follow the SW Requirements Specification content Signed-off-by: Sergey Vostokov * Add separated test items for testing of import Caffe/Caffe2 This commit adds new test items for each reqirements: - test import of Caffe NN model format - test import of Caffe2 NN model format Signed-off-by: Sergey Vostokov * Update reuirements mapping for test items This commit fixes wrong requirements mapping for test items Signed-off-by: Sergey Vostokov * Fix wrong numbering for test items This commit fixes wrong numbering for test item and for manual testing of functional requirement Signed-off-by: Sergey Vostokov * Remove RF-9::LLO from STD and fix typos This commit removes test item for RF-9:LLO as it is not presented in SRS Signed-off-by: Sergey Vostokov * Update revision of STD to 1.0 This commit updates revision of SW Test Document to Rev1.0 Signed-off-by: Sergey Vostokov --- doc/project/test_plan.md | 252 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 252 insertions(+) create mode 100644 doc/project/test_plan.md diff --git a/doc/project/test_plan.md b/doc/project/test_plan.md new file mode 100644 index 0000000..57f99b8 --- /dev/null +++ b/doc/project/test_plan.md @@ -0,0 +1,252 @@ +# SW System Test Document + +**Revision history** + +| Ver. | Date | Contents | Author | Approver | +| ---- | ---------- | -------------------------- | ------------------ | ------------ | +| 0.1 | 2018.04.12 | Initial version | Vostokov Sergey | Sung-Jae Lee | +| 0.2 | 2018.04.13 | SE member in-charge review | Aleksei Kondrashov | | +| 1.0 | 2018.04.17 | Final Execution DR version | Vostokov Sergey | Sung-Jae Lee | + +**Terminology and Abbreviation** + +| | | +| ------------ | ------------------------------------------------------------- | +| OS | Operating System | +| OS API | Application interface of OS | +| HW | Hardware | +| SW | Software | +| NN | Neural Network | +| NN model | Neural network model (Instance of NN built with ML framework) | +| NN compiler | The compiler for neural network | +| ML framework | The machine learning framework | +| TF/TF Lite | Tensorflow/Tensorflow Lite ML framework | +| IR | Intermediate representation | +| CI/CI system | Continuous integration system | +| UI | The user interface | +| GUI | The graphical user interface | +| CLI | The command-line interface | + +## SW System Test Overview + +### Purpose + +Software testing is an investigation to provide the quality of the +product under test and to reduce risk of its failure to users or +customers. Purpose of testing is to detect software failures so that +defects may be discovered and corrected. + +Software system test are processes and methods used to ensure quality. +Additional goals is to make sure the product follows regulations and +meets the quality standards expected by the customer. + +### Scope + +As the number of possible tests for every software is practically +infinite, we use some strategy to select tests that are feasible for the +available time and resources. + +Software system tests attempt to cover requirements listed in the [SW +Requirement +Specification](https://github.sec.samsung.net/STAR/nncc/doc/project/requirements_specification.md). + +Since the projest outcome is a compiler then its testing are in +different domain than many other kinds of application or system testing. +They are dedicated to find all possible issues that cause the following +bugs: + + - Compiler crashes (also known as an ICE or Internal Compiler Error) + - Compiler hangs (kind of infinite loop in the compiler) + - Bad code generation (a result of incorrect compiler output): + * Bad code generation that leads to a crash in the application + * “Silent” bad code generation + - Compiler throughput issues (Issues that affect the amount of time + the compiler takes to compile code ) + - Code quality issues (Issues that affect the performance of the + compiled application) + - Compiler feature correctness issues (This class of bugs involves the + compiler generating correct code, but not doing what a particular + feature specifies should be +done) + +## SW System Test Items + +### Functions to be tested + +| Feature | Test Item ID | Test Item description | +| ---------------------------------------- | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| RF-1, RIF-3 - RIF-7 | TST-1 | Test suite checks import of NN ops of Tensorflow Lite format: check loading NN model that consists of 1 NN ops. One test for 1 NN ops. | +| RF-2, RIF-3 - RIF-7 | TST-2 | Test suite checks import of NN ops of Caffe format: check loading NN model that consists of 1 NN ops. One test for 1 NN ops. | +| RF-3, RIF-3 - RIF-7 | TST-3 | Test suite checks import of NN ops of Caffe2 format: check loading NN model that consists of 1 NN ops. One test for 1 NN ops. | +| RF-5, RIF-3 - RIF-7 | TST-4 | The test should verify successful loading the Inception V3 NN model | +| RF-6, RIF-3 - RIF-7 | TST-5 | The test should verify successful loading the MobileNet NN model | +| RF-4 | TST-6 | The test suite should automatically verify the completeness of information which was read from the raw data using comparison them with serialized from Model IR raw data | +| RF-7, RF-19, RIF-13 | TST-7 | The unit test should automatically verify successful execution of binary on target ARM CPU | +| RF-8, RF-18, RIF-14, RIF-15 | TST-8 | The unit test should automatically verify successful execution of calculation on GPU | +| RF-10, RNF-1, RIF-17, RIF-18 | TST-9 | Unit test should verify the existence and format of binary (shared or static) in according to specified options. | +| RF-11 | TST-10 | Unit test should verify that compiler produce the compiled artefact for the Inception V3 NN model (Validity of compiled artefact is checked by other tests) | +| RF-12 | TST-11 | Unit test should verify that compiler produce the compiled artefact for the MobileNet NN model (Validity of compiled artefact is checked by other tests) | +| RF-13, RF-14, RF-15, RNF-6, RNF-7, RNF-8 | TST-12 | The test suite should verify correctness of configuration object by unit testing | +| RF-16, RNF-1 | TST-13 | The test suite is to verify the correctness of calculations by comparison the result of original NN model and result of compiled artefact on the same input date | +| RF-17 | TST-14 | Unit test should verify that the incorrect input data are processing with the error message without unexpected application termination | +| RNF-4, RNF-5, RIF-8 | TST-15 | The Linux-based OS should be used while the test environment are built. | +| RIF-16 | TST-16 | The unit test should verify the existence and validity of generated C/C++ header for compiled artefact | + +Table 2-1. Test Item + +**The following requirements can be tested only manually:** + + - Non-functional requirements: RNF-2, RNF-3 (They would be tested + during development) + - Interface requirements: RIF-1, RIF-2, RIF-9 - RIF-12, RIF-19 + +### Functions not to be tested + +The following requirements cannot be tested: + + - The source code requirements (RNF-9. RNF-10. RNF-11) + +## SW System Test Procedure + +### Test approaches + +While implementation of the project deliverables several kinds of +testing are used. All of them are performed automatically by continuous +integration system since it is developed. CI system subscribes on source +code modification in the version control system. The configuration does +not allow any changes to be merged into the main line if these changes +do not pass merge mandatory tests. + + - **Code style check** (Merge mandatory test): to verify consistency + of coding style + - **Build test** (Merge mandatory test): to verify the current build + - **Unit tests**: to verify SW system consistency. All new implemented + features, code refactoring, optimizations must not cause unit test + failure. Each unit test reflect the exact logic of testing + component, thus, it should be adopted any time when program logic + changes. + - **System tests**: to verify the feature quality as well as + compliance with its specified requirements. + - **Manual-based UI testing approach**: for interface requirements, + which cannot be automated + +### Test Pass/Fail Criteria + +All tests (unit/system) must be executed without any issues at any time +for newly implemented, refactored, or changed code. + +### Test Start/Suspension/Resumption criteria + +Two mandatory tests (code style check and build test) are performed for +every pool request (PR) before it is merged. The configuration of +continuous integration system (CI) does not allow to merge the changes +into devel branch if they does not pass the tests. + +Unit and feature testing are performed for the devel branch +automatically. The merge to master branch (release) are possible when +all these tests passed. + +### Regression Test strategy + +If a new issue is detected and it is not covered by an existing test +then a new test will be developed. In other case the issue should be +resolved. + +### Test tools + +| | | +| ------------------------------- | ------------------------------------------------------------------------------------ | +| Source code static verification | AEGIS (CODE pre-commit test suite: static/structure/open source violation analyzers) | +| Test execution | CMake | +| Defect management | Samsung Research GitHub | +| Continuous Integration system | HQ CI (CODE) | + +Table 3-1. Test Tools + +## SW System Test Schedule Plan + +### Test task & schedule + +| | | | | +| -------------- | ----------------------- | -------------- | -------------------------------------- | +| Task | Schedule | Responsibility | Detailed Task | +| Unit testing | 01.04.2018 - 31.12.2018 | All | All unit tests should be carried out | +| System testing | 01.04.2018 - 31.12.2018 | All | All system tests should be carried out | + +Table 4-1. Test Tasks and Schedule + +### Test Resource organization plan + +#### Test environment + +| Type/Model | Operating System | Usage | +| ---------- | --------------------------------- | ------------------------------------------------------------------------ | +| PC/x86 | Ubuntu GNU/Linux version \>=14.04 | Build system with unit tests. System and system tests are performed too. | +| Tizen TM2 | Tizen | Unit and system testing | +| Odroid XU4 | Tizen | Unit and system testing | + +Table 4-2. Hardware / Operating System + +| Type | Spec | Usage | +| ------------------- | ----------------------------------------------------- | ------------------------------------------------------------------------------- | +| Library | Google test | Organize test code and provide utility methods | +| VCS | Samsung github | The source code version controlling system | +| CI | CODE | The HQ CI system | +| Build system | CMake | Run test and check status | +| Device connectivity | sdb | Send tools to the device and provide shell to run it | +| Management tool | The CODE (Collaborative Open Development Environment) | Source code version control, code review, issue tracker, Continuous Integration | + +Table 4-3. Software + +### Risk management plan + +| Risk | Description | Probability | Countermeasures | +| ------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | ----------- | --------------------------------------------------------------------------------------- | +| SmartMachine OS SDK toolchain is not available | In order to support compilation for SmartMachine OS the SDK is required. The compiler would have dependency of a SmartMachine OS SDK toolchain. | High | Suspend support of SmartMachine OS, and make plans when SmartMachine OS SDK is released | +| SmartMachine OS targets are not available | To perform testing of executables for SmartMachine OS the specified targets are required. | High | Request targets or SW emulator when SmartMachine OS is released | +| HQ CI does not support target testing | Some tests required the target devices to be run on it. The provided CI system may not support such type of testing. | High | Set CI environment on site | +| Targets for testing/development are not available | Full automatic testing may take a long time. It also required target devices to execute the binaries. | Medium | Request/Buy enough amount of devices | + +Table 4-5. Risk Management + +### SW configuration management plan + +#### SW Configuration items identification + +| No | Document number | SW configuration Item | File name | +| -- | ------------------------- | ------------------------------ | ------------------------------------------- | +| 1 | SRR-RAJ0118ZZ-BWRF-STD001 | System Test Document | 18 NN compiler and Optimizer (STD) v1.0.pdf | +| 2 | SRR-RAJ0118ZZ-BWRF-STS001 | System Test Case Specification | 18 NN compiler and Optimizer (STS) v1.0.pdf | +| 3 | SRR-RAJ0118ZZ-BWRF-UTR001 | Unit Test Report | 18 NN compiler and Optimizer (UTR) v1.0.pdf | + +Table 4-6. SW Configuration Items List + +#### Directory Structure + +| Directory | Description | +| ------------------------ | ------------------------------------------------------------------ | +| / | source codes of the build system, main README file | +| /contrib | Incubating projects | +| /doc | Contains the documentation of the project | +| /libs | Contains the source of the libraries which are used by the nncc | +| /libs/core | Contains the source code of the core library of nncc | +| /libs/frontend | Contains the source code of supported frontend's plugins | +| /libs/frontend/caffe | The source code for the Caffe frontend | +| /libs/frontend/tflite | The source code for the Tensorflow Lite frontend | +| /libs/backend | Contains the source code of supported backend’ plugins | +| /libs/backend/cpu | Contains the source code of CPU backend | +| /libs/backend/gpu | Contains the source code of GPU backend | +| /libs/backend/3rd\_party | Contains the source code of backend to utilize 3rd party libraries | +| /scripts | Various scripts for building and testing the nncc | +| /tools | The source code of the executables | + +Table 4-7. Directory Structure + +#### Baseline + +| Test Round | Baseline Name | Configuration Item | Schedule | +| ---------- | ------------- | ---------------------------------------------------- | ---------- | +| Round 1 | The nncc v0.5 | SRR-RAJ0118ZZ-BWRF-STD001, SRR-RAJ0118ZZ-BWRF-UTR001 | 01.09.2018 | +| Round 2 | The nncc v1.0 | SRR-RAJ0118ZZ-BWRF-STD002, SRR-RAJ0118ZZ-BWRF-UTR002 | 01.12.2018 | + +Table 4-8. Baselines -- 2.7.4