--- /dev/null
+# SW System Test Document
+
+**Revision history**
+
+| Ver. | Date | Contents | Author | Approver |
+| ---- | ---------- | -------------------------- | ------------------ | ------------ |
+| 0.1 | 2018.04.12 | Initial version | Vostokov Sergey | Sung-Jae Lee |
+| 0.2 | 2018.04.13 | SE member in-charge review | Aleksei Kondrashov | |
+| 1.0 | 2018.04.17 | Final Execution DR version | Vostokov Sergey | Sung-Jae Lee |
+
+**Terminology and Abbreviation**
+
+| | |
+| ------------ | ------------------------------------------------------------- |
+| OS | Operating System |
+| OS API | Application interface of OS |
+| HW | Hardware |
+| SW | Software |
+| NN | Neural Network |
+| NN model | Neural network model (Instance of NN built with ML framework) |
+| NN compiler | The compiler for neural network |
+| ML framework | The machine learning framework |
+| TF/TF Lite | Tensorflow/Tensorflow Lite ML framework |
+| IR | Intermediate representation |
+| CI/CI system | Continuous integration system |
+| UI | The user interface |
+| GUI | The graphical user interface |
+| CLI | The command-line interface |
+
+## SW System Test Overview
+
+### Purpose
+
+Software testing is an investigation to provide the quality of the
+product under test and to reduce risk of its failure to users or
+customers. Purpose of testing is to detect software failures so that
+defects may be discovered and corrected.
+
+Software system test are processes and methods used to ensure quality.
+Additional goals is to make sure the product follows regulations and
+meets the quality standards expected by the customer.
+
+### Scope
+
+As the number of possible tests for every software is practically
+infinite, we use some strategy to select tests that are feasible for the
+available time and resources.
+
+Software system tests attempt to cover requirements listed in the [SW
+Requirement
+Specification](https://github.sec.samsung.net/STAR/nncc/doc/project/requirements_specification.md).
+
+Since the projest outcome is a compiler then its testing are in
+different domain than many other kinds of application or system testing.
+They are dedicated to find all possible issues that cause the following
+bugs:
+
+ - Compiler crashes (also known as an ICE or Internal Compiler Error)
+ - Compiler hangs (kind of infinite loop in the compiler)
+ - Bad code generation (a result of incorrect compiler output):
+ * Bad code generation that leads to a crash in the application
+ * “Silent” bad code generation
+ - Compiler throughput issues (Issues that affect the amount of time
+ the compiler takes to compile code )
+ - Code quality issues (Issues that affect the performance of the
+ compiled application)
+ - Compiler feature correctness issues (This class of bugs involves the
+ compiler generating correct code, but not doing what a particular
+ feature specifies should be
+done)
+
+## SW System Test Items
+
+### Functions to be tested
+
+| Feature | Test Item ID | Test Item description |
+| ---------------------------------------- | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| RF-1, RIF-3 - RIF-7 | TST-1 | Test suite checks import of NN ops of Tensorflow Lite format: check loading NN model that consists of 1 NN ops. One test for 1 NN ops. |
+| RF-2, RIF-3 - RIF-7 | TST-2 | Test suite checks import of NN ops of Caffe format: check loading NN model that consists of 1 NN ops. One test for 1 NN ops. |
+| RF-3, RIF-3 - RIF-7 | TST-3 | Test suite checks import of NN ops of Caffe2 format: check loading NN model that consists of 1 NN ops. One test for 1 NN ops. |
+| RF-5, RIF-3 - RIF-7 | TST-4 | The test should verify successful loading the Inception V3 NN model |
+| RF-6, RIF-3 - RIF-7 | TST-5 | The test should verify successful loading the MobileNet NN model |
+| RF-4 | TST-6 | The test suite should automatically verify the completeness of information which was read from the raw data using comparison them with serialized from Model IR raw data |
+| RF-7, RF-19, RIF-13 | TST-7 | The unit test should automatically verify successful execution of binary on target ARM CPU |
+| RF-8, RF-18, RIF-14, RIF-15 | TST-8 | The unit test should automatically verify successful execution of calculation on GPU |
+| RF-10, RNF-1, RIF-17, RIF-18 | TST-9 | Unit test should verify the existence and format of binary (shared or static) in according to specified options. |
+| RF-11 | TST-10 | Unit test should verify that compiler produce the compiled artefact for the Inception V3 NN model (Validity of compiled artefact is checked by other tests) |
+| RF-12 | TST-11 | Unit test should verify that compiler produce the compiled artefact for the MobileNet NN model (Validity of compiled artefact is checked by other tests) |
+| RF-13, RF-14, RF-15, RNF-6, RNF-7, RNF-8 | TST-12 | The test suite should verify correctness of configuration object by unit testing |
+| RF-16, RNF-1 | TST-13 | The test suite is to verify the correctness of calculations by comparison the result of original NN model and result of compiled artefact on the same input date |
+| RF-17 | TST-14 | Unit test should verify that the incorrect input data are processing with the error message without unexpected application termination |
+| RNF-4, RNF-5, RIF-8 | TST-15 | The Linux-based OS should be used while the test environment are built. |
+| RIF-16 | TST-16 | The unit test should verify the existence and validity of generated C/C++ header for compiled artefact |
+
+Table 2-1. Test Item
+
+**The following requirements can be tested only manually:**
+
+ - Non-functional requirements: RNF-2, RNF-3 (They would be tested
+ during development)
+ - Interface requirements: RIF-1, RIF-2, RIF-9 - RIF-12, RIF-19
+
+### Functions not to be tested
+
+The following requirements cannot be tested:
+
+ - The source code requirements (RNF-9. RNF-10. RNF-11)
+
+## SW System Test Procedure
+
+### Test approaches
+
+While implementation of the project deliverables several kinds of
+testing are used. All of them are performed automatically by continuous
+integration system since it is developed. CI system subscribes on source
+code modification in the version control system. The configuration does
+not allow any changes to be merged into the main line if these changes
+do not pass merge mandatory tests.
+
+ - **Code style check** (Merge mandatory test): to verify consistency
+ of coding style
+ - **Build test** (Merge mandatory test): to verify the current build
+ - **Unit tests**: to verify SW system consistency. All new implemented
+ features, code refactoring, optimizations must not cause unit test
+ failure. Each unit test reflect the exact logic of testing
+ component, thus, it should be adopted any time when program logic
+ changes.
+ - **System tests**: to verify the feature quality as well as
+ compliance with its specified requirements.
+ - **Manual-based UI testing approach**: for interface requirements,
+ which cannot be automated
+
+### Test Pass/Fail Criteria
+
+All tests (unit/system) must be executed without any issues at any time
+for newly implemented, refactored, or changed code.
+
+### Test Start/Suspension/Resumption criteria
+
+Two mandatory tests (code style check and build test) are performed for
+every pool request (PR) before it is merged. The configuration of
+continuous integration system (CI) does not allow to merge the changes
+into devel branch if they does not pass the tests.
+
+Unit and feature testing are performed for the devel branch
+automatically. The merge to master branch (release) are possible when
+all these tests passed.
+
+### Regression Test strategy
+
+If a new issue is detected and it is not covered by an existing test
+then a new test will be developed. In other case the issue should be
+resolved.
+
+### Test tools
+
+| | |
+| ------------------------------- | ------------------------------------------------------------------------------------ |
+| Source code static verification | AEGIS (CODE pre-commit test suite: static/structure/open source violation analyzers) |
+| Test execution | CMake |
+| Defect management | Samsung Research GitHub |
+| Continuous Integration system | HQ CI (CODE) |
+
+Table 3-1. Test Tools
+
+## SW System Test Schedule Plan
+
+### Test task & schedule
+
+| | | | |
+| -------------- | ----------------------- | -------------- | -------------------------------------- |
+| Task | Schedule | Responsibility | Detailed Task |
+| Unit testing | 01.04.2018 - 31.12.2018 | All | All unit tests should be carried out |
+| System testing | 01.04.2018 - 31.12.2018 | All | All system tests should be carried out |
+
+Table 4-1. Test Tasks and Schedule
+
+### Test Resource organization plan
+
+#### Test environment
+
+| Type/Model | Operating System | Usage |
+| ---------- | --------------------------------- | ------------------------------------------------------------------------ |
+| PC/x86 | Ubuntu GNU/Linux version \>=14.04 | Build system with unit tests. System and system tests are performed too. |
+| Tizen TM2 | Tizen | Unit and system testing |
+| Odroid XU4 | Tizen | Unit and system testing |
+
+Table 4-2. Hardware / Operating System
+
+| Type | Spec | Usage |
+| ------------------- | ----------------------------------------------------- | ------------------------------------------------------------------------------- |
+| Library | Google test | Organize test code and provide utility methods |
+| VCS | Samsung github | The source code version controlling system |
+| CI | CODE | The HQ CI system |
+| Build system | CMake | Run test and check status |
+| Device connectivity | sdb | Send tools to the device and provide shell to run it |
+| Management tool | The CODE (Collaborative Open Development Environment) | Source code version control, code review, issue tracker, Continuous Integration |
+
+Table 4-3. Software
+
+### Risk management plan
+
+| Risk | Description | Probability | Countermeasures |
+| ------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | ----------- | --------------------------------------------------------------------------------------- |
+| SmartMachine OS SDK toolchain is not available | In order to support compilation for SmartMachine OS the SDK is required. The compiler would have dependency of a SmartMachine OS SDK toolchain. | High | Suspend support of SmartMachine OS, and make plans when SmartMachine OS SDK is released |
+| SmartMachine OS targets are not available | To perform testing of executables for SmartMachine OS the specified targets are required. | High | Request targets or SW emulator when SmartMachine OS is released |
+| HQ CI does not support target testing | Some tests required the target devices to be run on it. The provided CI system may not support such type of testing. | High | Set CI environment on site |
+| Targets for testing/development are not available | Full automatic testing may take a long time. It also required target devices to execute the binaries. | Medium | Request/Buy enough amount of devices |
+
+Table 4-5. Risk Management
+
+### SW configuration management plan
+
+#### SW Configuration items identification
+
+| No | Document number | SW configuration Item | File name |
+| -- | ------------------------- | ------------------------------ | ------------------------------------------- |
+| 1 | SRR-RAJ0118ZZ-BWRF-STD001 | System Test Document | 18 NN compiler and Optimizer (STD) v1.0.pdf |
+| 2 | SRR-RAJ0118ZZ-BWRF-STS001 | System Test Case Specification | 18 NN compiler and Optimizer (STS) v1.0.pdf |
+| 3 | SRR-RAJ0118ZZ-BWRF-UTR001 | Unit Test Report | 18 NN compiler and Optimizer (UTR) v1.0.pdf |
+
+Table 4-6. SW Configuration Items List
+
+#### Directory Structure
+
+| Directory | Description |
+| ------------------------ | ------------------------------------------------------------------ |
+| / | source codes of the build system, main README file |
+| /contrib | Incubating projects |
+| /doc | Contains the documentation of the project |
+| /libs | Contains the source of the libraries which are used by the nncc |
+| /libs/core | Contains the source code of the core library of nncc |
+| /libs/frontend | Contains the source code of supported frontend's plugins |
+| /libs/frontend/caffe | The source code for the Caffe frontend |
+| /libs/frontend/tflite | The source code for the Tensorflow Lite frontend |
+| /libs/backend | Contains the source code of supported backend’ plugins |
+| /libs/backend/cpu | Contains the source code of CPU backend |
+| /libs/backend/gpu | Contains the source code of GPU backend |
+| /libs/backend/3rd\_party | Contains the source code of backend to utilize 3rd party libraries |
+| /scripts | Various scripts for building and testing the nncc |
+| /tools | The source code of the executables |
+
+Table 4-7. Directory Structure
+
+#### Baseline
+
+| Test Round | Baseline Name | Configuration Item | Schedule |
+| ---------- | ------------- | ---------------------------------------------------- | ---------- |
+| Round 1 | The nncc v0.5 | SRR-RAJ0118ZZ-BWRF-STD001, SRR-RAJ0118ZZ-BWRF-UTR001 | 01.09.2018 |
+| Round 2 | The nncc v1.0 | SRR-RAJ0118ZZ-BWRF-STD002, SRR-RAJ0118ZZ-BWRF-UTR002 | 01.12.2018 |
+
+Table 4-8. Baselines
projects / platform / core / ml / nnfw.git / commitdiff