--- /dev/null
+# SW Detailed Level Design
+
+**Revision history**
+
+| Ver. | Date | Contents | Author | Approver |
+| ---- | ---------- | ----------------- | ----------------- | ------------ |
+| 0.1 | 2018.06.20 | Initial version | Vostokov Sergey | Sung-Jae Lee |
+| 0.2 | 2018.06.21 | SE member review | Alexey Kondrashov | |
+| 1.0 | 2018.06.22 | Final DR1 version | Vostokov Sergey | Sung-Jae Lee |
+
+**Terminology and Abbreviation**
+
+| | |
+| ------------ | ------------------------------------------------------------- |
+| OS | Operating System |
+| OS API | Application interface of OS |
+| HW | Hardware |
+| SW | Software |
+| NN | Neural Network |
+| NN model | Neural network model (Instance of NN built with ML framework) |
+| NN compiler | The compiler for neural network |
+| ML framework | The machine learning framework |
+| TF/TF Lite | Tensorflow/Tensorflow Lite ML framework |
+| IR | Intermediate representation |
+| CI/CI system | Continuous integration system |
+| UI | The user interface |
+| GUI | The graphical user interface |
+| CLI | The command-line interface |
+
+**References**
+
+\[1\] Vostokov Sergey, [SW Requirements Specification](requirements_specification.md)
+
+\[2\] Vostokov Sergey, [SW High-Level Design](high_level_design.md)
+
+## Overview
+
+### Scope
+
+The main goal of the project is to develop a compiler for neural
+networks to produce executable artefact for specified SW and HW
+platform.
+
+The development scope includes the following components:
+
+ - Develop importer module to parse, verify and represent NN model for
+ further optimization and compilation
+ - Develop code emitters to produce executable binary for CPU and GPU
+
+
+**2018 year goals:**
+
+ - Support TensorFlow Lite NN model format
+ - Support Caffe NN model format
+ - Support Caffe2 NN model format (Optional)
+ - Support compilation of MobileNet NN
+ - Support compilation of Inception v3 NN
+ - Support ARM CPU
+ - Support ARM GPU (Mali)
+ - Support Tizen OS
+ - Support SmartMachine OS
+(Optional)
+
+| Product | Target Model Name | Comment |
+| ------------------- | ------------------------------ | ---------------- |
+| Tizen phone | Tizen TM2 | Reference device |
+| Tizen device | Odroid XU4 | Reference board |
+| SmartMachine target | Microvision mv8890, exynos8890 | Reference device |
+
+Table 1-1. Target Model
+
+### Design Consideration
+
+Deep learning software demands reliability and performance. The common
+approach which comes from the history is to develop a SW framework
+(machine learning framework) which would compute each step of the neural
+network inference process using supported hardware. This approach is
+used in many popular solutions like Google Tensorflow/Tensorflow Lite,
+Caffe/2, etc. Traditionally, neural network developers build a
+computation graph and then an appropriate machine learning framework
+interprets it. The latest discoveries in AI field show that the
+node-visitor method of execution is inefficient. As a result, a second
+approach has been worked out by the industry, which is a neural network
+compiler that executes code more efficiently.
+
+This document presents the design of the *nncc*, a neural network
+compiler collection. The design should provide the easiest way to extend
+the functionality of the *nncc* by adding new modules with the following
+features:
+
+ - Support neural networks produced by various machine learning
+ frameworks;
+ - Produce an artefact taking advantages of various hardware
+ including specialized processors like NPU;
+ - Apply new domain specific optimization techniques over given NN.
+
+Non-functional requirements to the developed software are well-described
+in the SW Requirements Specification, such requirements are not shown
+here to avoid duplication.
+
+### Constraints
+
+See constraints in SW Requirements Specification.
+
+
+<table>
+<colgroup>
+<col style="width: 24%" />
+<col style="width: 64%" />
+<col style="width: 10%" />
+</colgroup>
+<thead>
+<tr class="header">
+<th>Item</th>
+<th>Assumptions, Dependencies and the Constraints</th>
+<th>Reference</th>
+</tr>
+</thead>
+<tbody>
+<tr class="odd">
+<td>Tizen SW Platform</td>
+<td><dl>
+<dt>The following items should be provided:</dt>
+<dd><ul>
+<li>Tizen API</li>
+<li>Tizen kernel</li>
+<li>Tizen FW</li>
+<li>Tizen SDK</li>
+<li>Tizen naming convention</li>
+</ul>
+</dd>
+</dl></td>
+<td>- <a href="www.tizen.org" class="uri">www.tizen.org</a> <br>- <a href="wiki.tizen.org" class="uri">wiki.tizen.org</a> <br>- <a href="developer.tizen.org" class="uri">developer.tizen.org</a></td>
+</tr>
+<tr class="even">
+<td>SmartMachine OS Platform</td>
+<td><dl>
+<dt>The following items should be provided:</dt>
+<dd><ul>
+<li>SmartMachine API</li>
+<li>SmartMachine kernel</li>
+<li>SmartMachine FW</li>
+<li>SmartMachine SDK</li>
+<li>SmartMachine naming convention</li>
+</ul>
+</dd>
+</dl></td>
+<td>- <a href="http://suprem.sec.samsung.net/confluence/pages/viewpage.action?pageId=81833987">Platform confluence</a> <br>- <a href="https://github.sec.samsung.net/RS7-SmartMachine">Github</a> <br>- <a href="http://suprem.sec.samsung.net/confluence/display/ASEC/Adaptive+AUTOSAR">Functional Safety confluence</a></td>
+</tr>
+<tr class="odd">
+<td>Host OS</td>
+<td>Linux-based OS (Ubuntu, Archlinux, etc)</td>
+<td>- <a href="https://www.ubuntu.com/">Ubuntu site</a> <br>- <a href="https://www.archlinux.org/">Archlinux site</a></td>
+</tr>
+<tr class="even">
+<td>Tizen target HW</td>
+<td>The reference device should be provided: Tizen TM2</td>
+<td></td>
+</tr>
+<tr class="odd">
+<td>SmartMachine target HW</td>
+<td>The reference device should be provided</td>
+<td></td>
+</tr>
+</tbody>
+</table>
+Table 1-2. Assumptions, Dependecies and the Constraints</caption>
+
+## SW Detailed Structure Design
+
+### SW Block Structure
+
+Top-Level Components of the nncc descriped in HLD. More detailed
+structure and class diagram will be available after development
+completion.
+
+### SW Block Feature
+
+1. Initialization: configure all internal modules (see
+ [{Initialization} Detailed Design](#initialization-detailed-design))
+2. Frontend: Import NN model (see [{Import NN model} Detailed
+ Design](#import-nn-model-detailed-design))
+ - *Caffe frontend*: includes the parser of Caffe NN model format,
+ verifier to ensure that parsed data is valid and consentient,
+ and Caffe-specific IR converter
+ - *Caffe2 frontend*: includes the parser of Caffe2 NN model
+ format, verifier to ensure that parsed data is valid and
+ consentient, and Caffe2-specific IR converter to Model IR
+ - *Tensorflow Lite frontend*: includes the parser of Tensorflow NN
+ model format with automatic version recognition feature,
+ verifier to ensure that parsed data is valid and consentient,
+ and Tensorflow Lite-specific IR converter to Model IR
+3. Backend: Generate the code (see [{Generate the code} Detailed
+ Design](#generate-the-code-detailed-design))
+ - *Interpreter:* As it was described in SW High-Level Document
+ imported NN model may proceed through three step of Intermediate
+ representation: Model IR, Coarse-Grained IR, Fine-Grained IR.
+ The Interpreter backed uses each this IR to do inference of
+ given NN model. As the output, the user gets the resulting
+ calculation of all NN ops included into original computation
+ graph.
+ - *Binary*:This type refers to generating binary code that can be
+ executed on the target device. NN compiler can generate code
+ that is either executed solely on CPU or takes advantage of the
+ GPU when possible if the corresponding target was specified. The
+ user may want to incorporate 3rd party libraries included into
+ target firmware or delivered with the application package. In
+ this case, the compiler prepares the data following EABI
+ convention and embeds an invocation of high-level functions by
+ appropriate symbol.
+ - *Soft*: Resulting program is a generated source code in
+ high-level programming language C or C++. Here there are two
+ options: the first one is to generate the source code that does
+ not depend on libraries outside of itself, with the exception of
+ system libraries. The second one is to include the code to
+ invoke high-level functions from 3rd party libraries. For
+ example, it may be an invocation of matrix multiplication from
+ GEMM library.
+
+## SW Detailed Operation Design
+
+### {Initialization} Detailed Design
+
+#### Major Function
+
+To provide a valid configuration session for all modules of *nncc* using
+user input from the command line/config file/environment variables.
+
+#### Operation Sequence
+
+Initialization of the *nncc* includes command line option processing,
+configuration of its subsystems as well as any error checking possible
+at this stage. It consists of the following steps:
+
+1. Collect all command line options and verify their format for
+ validity (no syntax errors etc.)
+
+2. Check for validity and then process general options
+
+3. Load subsystem modules
+
+4. For each one of them:
+
+ - Configure
+ - Pass command line options
+ - Check command line options for validity (for example, check
+ that every required option is present)
+
+At the end of this process each subsystem is configured and has access
+to all data needed for its operation.
+
+### {Import NN model} Detailed Design
+
+#### Major Function
+
+To convert given NN model from framework-specific IR to Model IR for
+further processing.
+
+#### Operation Sequence
+
+As you may see on the diagram, neural network import is the main
+function of the compiler front-end part. The result of this operation is
+a computation graph which is presented as Model IR.
+
+
+
+The import process consists of three parts:
+
+1. NN model parsing
+2. Verification of the result from the previous step
+3. Converting the model to the Model IR
+
+During the first step, file or files containing the model are read and
+represented in some format specific to each NN framework.
+
+Verification step is included to ensure that:
+
+ - None of the files constituting the model are damaged
+ - Model format corresponds to the specified one
+ - Version of the model format corresponds to the specified one
+
+The most important step is accurately converting the model from the
+framework-specific representation to the Model IR. This conversion
+includes:
+
+ - *Translation of the NN model computation graph to the Model IR
+ computation graph.* During the translation new nodes may be
+ introduced - for example, a high-level NN operation may be split
+ into a few smaller ones.
+ - *NN model parameter layout conversion.* The way parameters (also
+ known as weights) of a model are layed out in each specific NN
+ framework may differ, and it is necessary to convert such layout
+ into a unified format.
+ - *NN operation parameter conversion.* Each NN operation has a set
+ of its own parameters describing the way this operation should be
+ performed, and these parameters also differ between frameworks.
+
+Resulting Model IR is equivalent to the initial NN model in terms of how
+NN model inputs would be transformed into its outputs if all the
+operations in the Model IR were executed.
+
+### {Generate the code} Detailed Design
+
+Development in progress. Will be described on Completion DR.
+
+## Interface Design
+
+Development in progress. Will be described on DR2.
+
+## SW Code Structure
+
+| Directory | Description |
+| ------------------------ | -------------------------------------------------------------------- |
+| / | source codes of the build system, main README file |
+| /contrib | Incubating projects |
+| /doc | Contains the documentation of the project |
+| /doc/project | Contains project management documents (SRS, SDD, STD, HLD, DLD, etc) |
+| /libs | Contains the source of the libraries which are used by the nncc |
+| /libs/core | Contains the source code of the core library of nncc |
+| /libs/frontend | Contains the source code of supported frontend's plugins |
+| /libs/frontend/caffe | The source code for the Caffe frontend |
+| /libs/frontend/caffe2 | The source code for the Caffe2 frontend |
+| /libs/frontend/tflite | The source code for the Tensorflow Lite frontend |
+| /libs/backend | Contains the source code of supported backend’ plugins |
+| /libs/backend/cpu | Contains the source code of CPU backend |
+| /libs/backend/gpu | Contains the source code of GPU backend |
+| /libs/backend/3rd\_party | Contains the source code of backend to utilize 3rd party libraries |
+| /scripts | Various scripts for building and testing the nncc |
+| /tools | The source code of the executables |
projects / platform / core / ml / nnfw.git / commitdiff