# NNtrainer
-[](http://nnsuite.mooo.com/nntrainer/ci/gcov_html/index.html)
+[](http://ci.nnstreamer.ai/nntrainer/ci/gcov_html/index.html)
+<a href="https://scan.coverity.com/projects/nnstreamer-nntrainer">
+ <img alt="Coverity Scan Build Status"
+ src="https://scan.coverity.com/projects/22512/badge.svg"/>
+</a>
+[](http://ci.nnstreamer.ai/nntrainer/ci/daily-build/build_result/)
-NNtrainer is Software Framework for Training Neural Network Models on Devices.
+NNtrainer is a Software Framework for training Neural Network models on devices.
## Overview
-NNtrainer is an Open Source Project. The aim of the NNtrainer is to develop Software Framework to train neural network model on embedded devices which has relatively limited resources. Rather than training the whole layers, NNtrainer trains only one or a few layers added after the feature extractor.
+NNtrainer is an Open Source Project. The aim of the NNtrainer is to develop a Software Framework to train neural network models on embedded devices which have relatively limited resources. Rather than training whole layers of a network from the scratch, NNtrainer finetunes the neural network model on device with user data for the personalization.
-Even though it trains part of the neural network models, NNtrainer requires quite a lot of functionalities to train from common neural network frameworks. By implementing them, it is good enough to run several examples which can help to understand how it works. There are k-NN, Neural Network, Logistic Regression and Reinforcement Learning with CartPole in Applications directory and some of them use Mobilenet V2 with tensorflow-lite as a feature extractor. All of them tested on Galaxy S8 with Android and PC (Ubuntu 16.04).
+Even if NNtariner runs on device, it provides full functionalities to train models and also utilizes limited device resources efficiently. NNTrainer is able to train various machine learning algorithms such as k-Nearest Neighbor (k-NN), Neural Networks, Logistic Regression, Reinforcement Learning algorithms, Recurrent network and more. We also provide examples for various tasks such as Few-shot learning, ResNet, VGG, Product Rating and more will be added. All of these were tested on Samsung Galaxy smart phone with Android and PC (Ubuntu 18.04/20.04).
+
+[ NNTrainer: Light-Weight On-Device Training Framework ](https://arxiv.org/pdf/2206.04688.pdf), arXiv, 2022 <br />
+[ NNTrainer: Towards the on-device learning for personalization ](https://www.youtube.com/watch?v=HWiV7WbIM3E), Samsung Software Developer Conference 2021 (Korean) <br />
+[ NNTrainer: Personalize neural networks on devices! ](https://www.youtube.com/watch?v=HKKowY78P1A), Samsung Developer Conference 2021 <br />
+[ NNTrainer: "On-device learning" ](https://www.youtube.com/embed/Jy_auavraKg?start=4035&end=4080), Samsung AI Forum 2021
+
+## Official Releases
+
+| | [Tizen](http://download.tizen.org/snapshots/tizen/unified/latest/repos/standard/packages/) | [Ubuntu](https://launchpad.net/~nnstreamer/+archive/ubuntu/ppa) | Android/NDK Build |
+| :-- | :--: | :--: | :--: |
+| | 6.0M2 and later | 18.04 | 9/P |
+| arm | [](http://ci.nnstreamer.ai/nntrainer/ci/daily-build/build_result/) | Available | Ready |
+| arm64 | [](http://ci.nnstreamer.ai/nntrainer/ci/daily-build/build_result/) | Available | [](http://ci.nnstreamer.ai/nntrainer/ci/daily-build/build_result/) |
+| x64 | [](http://ci.nnstreamer.ai/nntrainer/ci/daily-build/build_result/) | [](http://ci.nnstreamer.ai/nntrainer/ci/daily-build/build_result/) | Ready |
+| x86 | [](http://ci.nnstreamer.ai/nntrainer/ci/daily-build/build_result/) | N/A | N/A |
+| Publish | [Tizen Repo](http://download.tizen.org/snapshots/tizen/unified/latest/repos/standard/packages/) | [PPA](https://launchpad.net/~nnstreamer/+archive/ubuntu/ppa) | |
+| API | C (Official) | C/C++ | C/C++ |
+
+- Ready: CI system ensures build-ability and unit-testing. Users may easily build and execute. However, we do not have automated release & deployment system for this instance.
+- Available: binary packages are released and deployed automatically and periodically along with CI tests.
+- [Daily Release](http://ci.nnstreamer.ai/nntrainer/ci/daily-build/build_result/)
+- SDK Support: Tizen Studio (6.0 M2+)
## Maintainer
* [Jijoong Moon](https://github.com/jijoongmoon)
* [Gichan Jang](https://github.com/gichan-jang)
* [Yongjoo Ahn](https://github.com/anyj0527)
* [Jihoon Lee](https://github.com/zhoonit)
+* [Hyeonseok Lee](https://github.com/lhs8928)
+* [Mete Ozay](https://github.com/meteozay)
+* [Hyunil Park](https://github.com/songgot)
+* [Jiho Chu](https://github.com/jihochu)
+* [Yelin Jeong](https://github.com/niley7464)
+* [Donghak Park](https://github.com/DonghakPark)
+
## Components
### Supported Layers
-This component defines Layers which consist of Neural Network Model. Layers has own properties to be set.
+This component defines layers which consist of a neural network model. Layers have their own properties to be set.
- | Keyword | Layer Name | Description |
+ | Keyword | Layer Class Name | Description |
|:-------:|:---:|:---|
- | conv2d | Convolution 2D |Convolution 2-Dimentional Layer |
- | pooling2d | Pooling 2D |Pooling 2-Dimentional Layer. Support average / max / global average / global max pooing |
- | flatten | Flatten | Flatten Layer |
- | fully_connected | Fully Connected | Fully Connected Layer |
- | input | Input | Input Layer. This is not always requied. |
- | batch_normalization | Batch Normalization Layer | Batch Normalization Layer. |
- | loss layer | loss layer | hidden from users |
- | activation | activaiton layer | set by layer property |
+ | conv1d | Conv1DLayer | Convolution 1-Dimentional Layer |
+ | conv2d | Conv2DLayer |Convolution 2-Dimentional Layer |
+ | pooling2d | Pooling2DLayer |Pooling 2-Dimentional Layer. Support average / max / global average / global max pooling |
+ | flatten | FlattenLayer | Flatten layer |
+ | fully_connected | FullyConnectedLayer | Fully connected layer |
+ | pooling2D | Pooling2DLayer | Pooling 2D layer |
+ | input | InputLayer | Input Layer. This is not always required. |
+ | batch_normalization | BatchNormalizationLayer | Batch normalization layer |
+ | layer_normalization | LayerNormalizationLayer | Layer normalization layer |
+ | activation | ActivaitonLayer | Set by layer property |
+ | addition | AdditionLayer | Add input input layers |
+ | attention | AttentionLayer | Attenstion layer |
+ | centroid_knn | CentroidKNN | Centroid K-nearest neighbor layer |
+ | concat | ConcatLayer | Concatenate input layers |
+ | multiout | MultiOutLayer | Multi-Output Layer |
+ | backbone_nnstreamer | NNStreamerLayer | Encapsulate NNStreamer layer |
+ | backbone_tflite | TfLiteLayer | Encapsulate tflite as a layer |
+ | permute | PermuteLayer | Permute layer for transpose |
+ | preprocess_flip | PreprocessFlipLayer | Preprocess random flip layer |
+ | preprocess_l2norm | PreprocessL2NormLayer | Preprocess simple l2norm layer to normalize |
+ | preprocess_translate | PreprocessTranslateLayer | Preprocess translate layer |
+ | reshape | ReshapeLayer | Reshape tensor dimension layer |
+ | split | SplitLayer | Split layer |
+ | dropout | DropOutLayer | Dropout Layer |
+ | embedding | EmbeddingLayer | Embedding Layer |
+ | positional_encoding | PositionalEncodingLayer | Positional Encoding Layer |
+ | rnn | RNNLayer | Recurrent Layer |
+ | rnncell | RNNCellLayer | Recurrent Cell Layer |
+ | gru | GRULayer | Gated Recurrent Unit Layer |
+ | grucell | GRUCellLayer | Gated Recurrent Unit Cell Layer |
+ | lstm | LSTMLayer | Long Short-Term Memory Layer |
+ | lstmcell | LSTMCellLayer | Long Short-Term Memory Cell Layer |
+ | zoneoutlstmcell | ZoneoutLSTMCellLayer | Zoneout Long Short-Term Memory Cell Layer |
+ | time_dist | TimeDistLayer | Time distributed Layer |
+ | multi_head_attention | MultiHeadAttentionLayer | Multi Head Attention Layer |
+
### Supported Optimizers
NNTrainer Provides
- | Keyward | Optimizer Name | Description |
+ | Keyword | Optimizer Name | Description |
|:-------:|:---:|:---:|
| sgd | Stochastic Gradient Decent | - |
| adam | Adaptive Moment Estimation | - |
-### Supported Loss
+ | Keyword | Leanring Rate | Description |
+ |:-------:|:---:|:---:|
+ | exponential | exponential learning rate decay | - |
+ | constant | constant learning rate | - |
+ | step | step learning rate | - |
+
+### Supported Loss Functions
NNTrainer provides
- | Keyward | Loss Name | Description |
+ | Keyword | Class Name | Description |
|:-------:|:---:|:---:|
- | mse | Mean squared Error | - |
- | cross | Cross Entropy - sigmoid | if activation last layer is sigmoid |
- | cross | Cross Entropy - softmax | if activation last layer is softmax |
+ | cross_sigmoid | CrossEntropySigmoidLossLayer | Cross entropy sigmoid loss layer |
+ | cross_softmax | CrossEntropySoftmaxLossLayer | Cross entropy softmax loss layer |
+ | constant_derivative | ConstantDerivativeLossLayer | Constant derivative loss layer |
+ | mse | MSELossLayer | Mean square error loss layer |
+ | kld | KLDLossLayer | Kullback-Leibler Divergence loss layer |
-### Supported Activations
+### Supported Activation Functions
NNTrainer provides
- | Keyward | Loss Name | Description |
+ | Keyword | Loss Name | Description |
|:-------:|:---:|:---|
| tanh | tanh function | set as layer property |
| sigmoid | sigmoid function | set as layer property |
| relu | relu function | set as layer propery |
| softmax | softmax function | set as layer propery |
- | weight_initializer | Weight Initialization | Xavier(Normal/Uniform), LeCun(Normal/Uniform), HE(Normal/Unifor) |
- | weight_regularizer | weight decay ( L2Norm only ) | needs set weight_regularizer_param & type |
- | learnig_rate_decay | learning rate decay | need to set step |
### Tensor
-Tensor is responsible for the calculation of Layer. It executes the addition, division, multiplication, dot production, averaging of Data and so on. In order to accelerate the calculation speed, CBLAS (C-Basic Linear Algebra: CPU) and CUBLAS (CUDA: Basic Linear Algebra) for PC (Especially NVIDIA GPU) for some of the operation. Later, these calculations will be optimized.
-Currently we supports lazy calculation mode to reduce copy of tensors during calcuation.
+Tensor is responsible for calculation of a layer. It executes several operations such as addition, division, multiplication, dot production, data averaging and so on. In order to accelerate calculation speed, CBLAS (C-Basic Linear Algebra: CPU) and CUBLAS (CUDA: Basic Linear Algebra) for PC (Especially NVIDIA GPU) are implemented for some of the operations. Later, these calculations will be optimized.
+Currently, we supports lazy calculation mode to reduce complexity for copying tensors during calculations.
- | Keyward | Description |
+ | Keyword | Description |
|:-------:|:---:|
| 4D Tensor | B, C, H, W|
| Add/sub/mul/div | - |
NNTrainer provides
- | Keyward | Loss Name | Description |
+ | Keyword | Loss Name | Description |
|:-------:|:---:|:---|
| weight_initializer | Weight Initialization | Xavier(Normal/Uniform), LeCun(Normal/Uniform), HE(Normal/Unifor) |
- | weight_regularizer | weight decay ( L2Norm only ) | needs set weight_regularizer_constant & type |
- | learnig_rate_decay | learning rate decay | need to set step |
+ | weight_regularizer | weight decay ( L2Norm only ) | needs set weight_regularizer_param & type |
### APIs
-Currently we privde [C APIs](https://github.com/nnstreamer/nntrainer/blob/master/api/capi/include/nntrainer.h) for Tizen. C++ API also provides soon.
-
-
-### Examples for NNTrainer
-
-#### [Custom Shortcut Application](https://github.com/nnstreamer/nntrainer/tree/master/Applications/Tizen_native/CustomShortcut)
-
-
-This is demo application which enable user defined custom shortcut on galaxy watch.
-
-#### [MNIST Example](https://github.com/nnstreamer/nntrainer/tree/master/Applications/MNIST)
-
-This is example to train mnist dataset. It consists two convolution 2d layer, 2 pooling 2d layer, flatten layer and fully connected layer.
-
-#### [Reinforcement Learning Example](https://github.com/nnstreamer/nntrainer/tree/master/Applications/ReinforcementLearning/DeepQ)
-
-This is reinforcement learning example with cartpole game. It is using deepq alogrightm.
+Currently, we provide [C APIs](https://github.com/nnstreamer/nntrainer/blob/master/api/capi/include/nntrainer.h) for Tizen. [C++ APIs](https://github.com/nnstreamer/nntrainer/blob/master/api/ccapi/include) are also provided for other platform. Java & C# APIs will be provided soon.
-#### [Classification for cifar 10](https://github.com/nnstreamer/nntrainer/tree/master/Applications/TransferLearning/CIFAR_Classification)
-This is Transfer learning example with cifar 10 data set. TFlite is used for feature extractor and modify last layer (fully connected layer) of network.
+## [Getting Started](https://github.com/nnstreamer/nntrainer/blob/main/docs/getting-started.md)
-#### ~Tizen CAPI Example~
+Instructions for installing NNTrainer.
-~This is for demonstrate c api for tizen. It is same transfer learing but written with tizen c api.~
-Deleted instead moved to a [test](https://github.com/nnstreamer/nntrainer/blob/master/test/tizen_capi/unittest_tizen_capi.cpp)
+### [Running Examples](https://github.com/nnstreamer/nntrainer/blob/main/docs/how-to-run-examples.md)
-#### [KNN Example](https://github.com/nnstreamer/nntrainer/tree/master/Applications/KNN)
+Instructions for preparing NNTrainer for execution
-This is Transfer learning example with cifar 10 data set. TFlite is used for feature extractor and compared with KNN
+### [Examples for NNTrainer](https://github.com/nnstreamer/nntrainer/tree/main/Applications)
-#### [Logistic Regression Example](https://github.com/nnstreamer/nntrainer/tree/master/Applications/LogisticRegression)
+NNTrainer example for a variety of networks
-This is simple logistic regression example using nntrainer.
-
-## Getting Started
-
-### Prerequisites
-
-The following dependencies are needed to compile / build / run.
-
-* gcc/g++ (>= 4.9, std=c++14 is used)
-* meson (>= 0.50.0)
-* blas library (CBLAS) (for CPU Acceleration, libopenblas is used for now)
-* cuda, cudart, cublas (should match the version) (GPU Acceleration on PC)
-* tensorflow-lite (>= 1.4.0)
-* libjsoncpp ( >= 0.6.0) (openAI Environment on PC)
-* libcurl3 (>= 7.47) (openAI Environment on PC)
-* libiniparser
-* libgtest (for testing)
-
-
-### Give It a Go Build with Docker
-
-You can use [docker image](https://hub.docker.com/r/lunapocket/nntrainer-build-env) to easily set up and try building.
-
-To run the docker
-
-```bash
-$ docker pull lunapocket/nntrainer-build-env:ubuntu-18.04
-$ docker run --rm -it lunapocket/nntrainer-build-env:ubuntu-18.04
-```
-
-Inside docker...
-
-```bash
-$ cd /root/nntrainer
-$ git pull # If you want to build with latest sources.
-```
-
-You can try build from now on without concerning about Prerequisites.
-
-### How to Build
-
-Download the source file by cloning the github repository.
-
-```bash
-$ git clone https://github.com/nnstreamer/nntrainer
-```
-
-After completing download the sources, you can find the several directories and files as below.
-
-``` bash
-$ cd nntrainer
+## Open Source License
-$ ls -1
-api
-Applications
-debian
-doc
-jni
-LICENSE
-meson.build
-meson_options.txt
-nntrainer
-nntrainer.pc.in
-packaging
-README.md
-test
+The NNtrainer is an open source project released under the terms of the Apache License version 2.0.
-$ git log --oneline
-f1a3a05 (HEAD -> master, origin/master, origin/HEAD) Add more badges
-37032a1 Add Unit Test Cases for Neural Network Initialization
-181a003 lower case for layer type.
-1eb399b Update clang-format
-87f1de7 Add Unit Test for Neural Network
-cd5c36e Add Coverage Test badge for nntrainer
-...
-```
-
-You can find the source code of the core library in nntrainer/src. In order to build them, use [meson](https://mesonbuild.com/)
-```bash
-$ meson build
-The Meson build system
-Version: 0.50.1
-Source dir: /home/wook/Work/NNS/nntrainer
-Build dir: /home/wook/Work/NNS/nntrainer/build
-Build type: native build
-Project name: nntrainer
-Project version: 0.0.1
-Native C compiler: cc (gcc 7.5.0 "cc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0")
-Native C++ compiler: c++ (gcc 7.5.0 "c++ (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0")
-Build machine cpu family: x86_64
-Build machine cpu: x86_64
-...
-Build targets in project: 11
-Found ninja-1.8.2 at /usr/bin/ninja
-
-$ ninja -C build
-ninja: Entering directory `build'
-[41/41] Linking target test/unittest/unittest_nntrainer_internal.
-```
-
-After completion of the build, the shared library, 'libnntrainer.so' and the static library, 'libnntrainer.a' will be placed in build/nntrainer.
-```bash
-$ ls build/nntrainer -1
-d48ed23@@nntrainer@sha
-d48ed23@@nntrainer@sta
-libnntrainer.a
-libnntrainer.so
-```
-
-In order to install them with related header files to your system, use the 'install' sub-command.
-```bash
-$ ninja -C build install
-```
-Then, you will find the libnntrainer.so and related .h files in /usr/local/lib and /usr/local/include directories.
-
-By default, the command ```ninja -C build`` generates the five example application binaries (Classification, k-NN, LogisticRegression, ReinforcementLearning, and Training) you could try in build/Applications. For 'Training' as an example case,
-```bash
-$ ls build/Applications/Training/jni/ -1
-e189c96@@nntrainer_training@exe
-nntrainer_training
-```
-
-In order to run such example binaries, Tensorflow-lite is a prerequisite. If you are trying to run on the Android, it will automatically download tensorflow (1.9.0) and compile as static library. Otherwise, you need to install it by yourself.
-
-### How to Test
-
-1. Unittest
-Meson build `enable-test` set to true
-
-```bash
-$ echo $(pwd)
-(project root)
-
-$ meson build -Denable-test=true
-The Meson build system
-Version: 0.54.3
-...
-Configuring capi-nntrainer.pc using configuration
-Run-time dependency GTest found: YES (building self)
-Build targets in project: 17
-
-Found ninja-1.10.0.git.kitware.jobserver-1 at /home/jlee/.local/bin/ninja
-
-$ ninja -C build test
-[79/79] Running all tests.
- 1/12 unittest_tizen_capi OK 8.86s
- 2/12 unittest_tizen_capi_layer OK 0.05s
- 3/12 unittest_tizen_capi_optimizer OK 0.01s
- 4/12 unittest_tizen_capi_dataset OK 0.03s
- 5/12 unittest_nntrainer_activations OK 0.03s
- 6/12 unittest_nntrainer_internal OK 0.23s
- 7/12 unittest_nntrainer_layers OK 0.22s
- 8/12 unittest_nntrainer_lazy_tensor OK 0.04s
- 9/12 unittest_nntrainer_tensor OK 0.04s
-10/12 unittest_util_func OK 0.05s
-11/12 unittest_databuffer_file OK 0.12s
-12/12 unittest_nntrainer_modelfile OK 2.22s
-
-Ok: 12
-Expected Fail: 0
-Fail: 0
-Unexpected Pass: 0
-Skipped: 0
-Timeout: 0
-```
-
-if you want to run particular test
-
-```bash
-$ meson -C build test $(test name)
-```
-
-2. Sample app test
-NNTrainer provides extensive sample app running test.
-
-Meson build with `enable-app` set to true
-```bash
-$ echo $(pwd)
-(project root)
-
-$ meson build -Denable-app=true
-The Meson build system
-Version: 0.54.3
-...
-Configuring capi-nntrainer.pc using configuration
-Run-time dependency GTest found: YES (building self)
-Build targets in project: 17
-
-Found ninja-1.10.0.git.kitware.jobserver-1 at /home/jlee/.local/bin/ninja
-
-$ ninja -C build test
-...
- 1/21 app_classification OK 3.59s
- 2/21 app_classification_func OK 42.77s
- 3/21 app_knn OK 4.81s
- 4/21 app_logistic OK 14.11s
- 5/21 app_DeepQ OK 30.30s
- 6/21 app_training OK 38.36s
- 7/21 app_classification_capi_ini OK 32.65s
- 8/21 app_classification_capi_file OK 32.04s
- 9/21 app_classification_capi_func OK 29.13s
- ...
-```
-
-if you want to run particular example only
-
-```bash
-$ meson -C build test $(test name) #app_classification_capi_func
-```
-
-### Running Examples
-
-1. [Training](https://github.com/nnstreamer/nntrainer/blob/master/Applications/Training/README.md)
-
-After build, run with following arguments
-Make sure to put last '/' for the resources directory.
-```bash
-$./path/to/example ./path/to/settings.ini ./path/to/resource/directory/
-```
-
-To run the 'Training', for example, do as follows.
-
-```bash
-$ pwd
-./nntrainer
-$ LD_LIBRARY_PATH=./build/nntrainer ./build/Applications/Training/jni/nntrainer_training ./Applications/Training/res/Training.ini ./Applications/Training/res/
-../../res/happy/happy1.bmp
-../../res/happy/happy2.bmp
-../../res/happy/happy3.bmp
-../../res/happy/happy4.bmp
-../../res/happy/happy5.bmp
-../../res/sad/sad1.bmp
-../../res/sad/sad2.bmp
-
-...
-
-```
+## Contributing
-## Open Source License
+Contributions are welcome! Please see our [Contributing](https://github.com/nnstreamer/nntrainer/blob/main/docs/contributing.md) Guide for more details.
-The nntrainer is an open source project released under the terms of the Apache License version 2.0.
+[](https://sourcerer.io/fame/dongju-chae/nnstreamer/nntrainer/links/0)[](https://sourcerer.io/fame/dongju-chae/nnstreamer/nntrainer/links/1)[](https://sourcerer.io/fame/dongju-chae/nnstreamer/nntrainer/links/2)[](https://sourcerer.io/fame/dongju-chae/nnstreamer/nntrainer/links/3)[](https://sourcerer.io/fame/dongju-chae/nnstreamer/nntrainer/links/4)[](https://sourcerer.io/fame/dongju-chae/nnstreamer/nntrainer/links/5)[](https://sourcerer.io/fame/dongju-chae/nnstreamer/nntrainer/links/6)[](https://sourcerer.io/fame/dongju-chae/nnstreamer/nntrainer/links/7)
projects / platform / core / ml / nntrainer.git / blobdiff