[layercontext] Minor bugfix for layer context

Minor bugfix for layer context
Removes the extra trainable parameter from layer context

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[networkgraph] Network graph updated for Layer V2

Network graph updated to work with LayerV2
This involves settings input dimension in InitContext, and setting
up RunContext for each layer before their execution.

Further, some helper functions are also added in LayerNode.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[Test/Prepare] Add layer semantics test

For layer_v2, there will be more structured way to access tests.
This patch adds a skeleton for layer semantics test.

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Layer] Add trainable prop

This patch add trainable property to layer

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[manager] Add support for request Inputs/outputs

Added support for requesting inputs and outputs
Also update implementation to use a common function for easier
management

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[layer-internal] refactoring weight, input, output to layer_node

- Implement getter for variable and grad of weight, input, output
- Implement getNumWeights

Self evaluation:

Build test: [X]Passed [ ]Failed [ ]Skipped
Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: hyeonseok lee <hs89.lee@samsung.com>

[graph] Support creating RunLayerContext

Implement the function to create RunLayerContext given the
InitLayerContext and update it in the given LayerNode.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[manager] Add support for request Inputs/outputs

Added support for requesting inputs and outputs
Also update implementation to use a common function for easier
management

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[Properties] Add float and boolean cases

Add float and boolean case to be used later.

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Props] Add concept of empty to property

This patch adds empty concept to property instead of having insensible
value inside a property as a default

before this patch,
```cpp
auto a = Property<int>();
EXPECT_EQ(a.get(), -1); // let's assume -1 is default value which might not be sensible.
```
after this patch

```cpp
auto a = Property<int>();
EXPECT_THROW(a.get(), std::invalid_argument);
EXPECT_TRUE(a.empty());
```

so underlying value remains always null or valid.

Now, it became natural to distinguish mandatory props and non-mandatory
props

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Layer] Scaffold layer impl

**Major changes**
- Create `LayerImpl`
- Change `LayerDevel` signature
  s/int initialize/void finalize
  s/int setProperty/void setProperty
- Move createLayer to layerImpl

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Layer] Add const to the layer::getName()

getName() is independent of const, so this patch adds const to
layer::getName();

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[layer-internal] refactoring getInputDimension getOutputDimension

implement getInputDimension, getOutputDimension in layer_node

Self evaluation:

Build test: [X]Passed [ ]Failed [ ]Skipped
Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: hyeonseok lee <hs89.lee@samsung.com>

[neuralnet] Bug fix for graph usage

Bug fix patch to use constant iterators for the graph.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[graph] Remove non-const iterators

Remove non-const iterators for the graph.
Update the corresponding usages for the graph.

Note: the pattern for (auto const &node : graph) has been
removed as this can begin()/end() in the background.
Workaround for this is to use std::as_const(graph) but this
is only available from c++17, so this patch uses
the basic const iterator for loop pattern.

**Self evaluation:**
1. Build test: [x]Passed [ ]Failed [ ]Skipped
2. Run test: [x]Passed [ ]Failed [ ]Skipped

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[graph] Graph iterator for sorted and unsorted

Updated graph iteartor to run over unsorted list of nodes
if the graph is not yet sorted. If the graph is sorted,
the iterator iterates over sorted nodes.

**Self evaluation:**
1. Build test: [x]Passed [ ]Failed [ ]Skipped
2. Run test: [x]Passed [ ]Failed [ ]Skipped

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[graph] Update node_names to unordered_set

Update node_names storing all the names of the nodes
in the graph for uniqueness to use unordered_set.

**Self evaluation:**
1. Build test: [x]Passed [ ]Failed [ ]Skipped
2. Run test: [x]Passed [ ]Failed [ ]Skipped

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[graph] Make adjacency list lazily

Update graph to make adjacenct list lazily when required
during the topological sort.
Until then, keep the node_list which contains simply all the nodes.

**Self evaluation:**
1. Build test: [x]Passed [ ]Failed [ ]Skipped
2. Run test: [x]Passed [ ]Failed [ ]Skipped

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[manager] Support request Tensors and Weights

Added support for request Tensors and Weights
Moved spec of Tensors and Weights to VarGrad and Weights correspondingly
Also created constructors for weights and var_grad with their specs

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[layer_internal] refactoring read, save

implement read, save function in layer_node

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: hyeonseok lee <hs89.lee@samsung.com>

[Test/Scaffolding] Prepare test for the new layer

For now, layer test is too cluttered so that it is not agile enought to
make any change.

This PR mainly separates the tests into two parts to make the test
scallable itself.

1. Layer only test leaving inside `unittest_layers_{layer_name}`
2. Common tests like setProperty, golden_test in `layers_common_tests.h`

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Refactor] s/Layer/LayerV1

Change class name of layer to layerV1 to prepare migration
This should pass CI test.

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[context] Layer context creation scaffolding

Added todo list for the layer context creation
Added minor new interfaces for the layer context
Disable in-place optimization and inference memory optimization

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[addition] Bug fix addition layer calcDerivative

Bug fix for addition layer calcDerivative().
addition layer assign the same tensor derivative memory back to its
other layers. If the other layers connecting with addition layers are
inplace, then this can lead to wrong results.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[Plugin/Test] Add preset plugin for general test

This extracts common test to be used multiple times.

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Custom/Loss] Add scaffolding for loss example

This patch generates a skeleton code for mae custom loss example.

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[CC17] Update type to inline static

This patch update `type` to be inline static data member inside an
object for clarity

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[ GRU ] Add GRU Unittest

This commit includes,
  . unittests of gru layer
  . add keras code to compare

**Self evaluation:**
1. Build test:  [X]Passed [ ]Failed [ ]Skipped
2. Run test:  [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: jijoong.moon <jijoong.moon@samsung.com>

[ Recurrent ] Fix activation type

using hidden_state_activation_type

**Self evaluation:**
1. Build test:  [X]Passed [ ]Failed [ ]Skipped
2. Run test:  [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: jijoong.moon <jijoong.moon@samsung.com>

[ GRU ] implement fowarding / backwarding of GRU Layer

This commit includes,
  . forwarding implementation
  . calGradient implementation
  . calDerivative implementation
  . gru_basic unittest

**Self evaluation:**
1. Build test:  [X]Passed [ ]Failed [ ]Skipped
2. Run test:  [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: jijoong.moon <jijoong.moon@samsung.com>

[Props] Add concept of empty to property

This patch adds empty concept to property instead of having insensible
value inside a property as a default

before this patch,
```cpp
auto a = Property<int>();
EXPECT_EQ(a.get(), -1); // let's assume -1 is default value which might not be sensible.
```
after this patch

```cpp
auto a = Property<int>();
EXPECT_THROW(a.get(), std::invalid_argument);
EXPECT_TRUE(a.empty());
```

so underlying value remains always null or valid.

Now, it became natural to distinguish mandatory props and non-mandatory
props

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[ Unit Test ] Fix LSTM / RNN Unittest

This commit includes,
  . fix wrong unittest cases of lstm & rnn

**Self evaluation:**
1. Build test:  [X]Passed [ ]Failed [ ]Skipped
2. Run test:  [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: jijoong.moon <jijoong.moon@samsung.com>

[ GRU ] Skeleton Code for GRU

This commit includes,
 . Skeleton Code for GRU Layer

**Self evaluation:**
1. Build test:  [X]Passed [ ]Failed [ ]Skipped
2. Run test:  [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: jijoong.moon <jijoong.moon@samsung.com>

[docs] Update docs for tflite version

Updated docs with tensorflow-lite version to be 2.3.0 or higher
and added missing dependencies for local meson build as well.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[Refactor] s/Layer/LayerV1

Change class name of layer to layerV1 to prepare migration
This should pass CI test.

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[API/Devel] Retouch devel header

**Changes proposed in this PR:**
- Change function sinature to exploit forward declaration
- s/export_to/exportTo
- Change header guard to be unique

**Self evaluation:**
1. Build test: [ ]Passed [ ]Failed [X]Skipped
2. Run test: [ ]Passed [ ]Failed [X]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[neuralnet] Neural network bugfix

Two PRs merged have caused main branch to fail.
This patch provides temporary fix.
Proper fix  will be done with layer_v2.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[unittests] Update unittests for new gtest version

Remove gtest version check comments and update unittests for the new
gtest versions.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[Loss] Apply requireLabel to feed label

Currently, loss layer is the only one that can accept label which was
prohibiting building a custom loss layer.

**Major Changes**
- Apply requireLabel to feed label
- Last layer is not assumed loss layer when feeding label(still it is in
other places, need to be dealt)

**Minor Changes**
- Feeding multiple label is now possible (spliting label is future work)
- [Fix] Count Trainable includes loss layer
- [Fix] Handle cases When all layer is not trainable for
neuralnet::backwarding

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[layer] Move name to LayerNode

Move layer identifier name to LayerNode.
Now each node in the graph has name as its direct identifier.

This patch also disable unittest_nntrianer_layers for some while
till layer refactoring is taking place. However, other unittests
and apptests must remain enabled to ensure that everything else is
working.

**Self evaluation:**
1. Build test: [x]Passed [ ]Failed [ ]Skipped
2. Run test: [x]Passed [ ]Failed [ ]Skipped

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[split layer] Initialize class member bugfix

Split layer has class member leading_helper_dim
which remains uninitialized and results in bugs
when setBatchSize() is called before initialization of the layer.

This patch fixes the bug.

**Self evaluation:**
1. Build test: [x]Passed [ ]Failed [ ]Skipped
2. Run test: [x]Passed [ ]Failed [ ]Skipped

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[layer context] Propose layer context and layer_devel

This patch proposes layer context and layer_devel.
The layer context provides basic necessities for each layer
like inputs, outputs, weights, tensors. Weights are the trainable tensors
required by the layers which can be requested with the context interfaces.
Correspondingly, any other tensor requirements by the layer can also be
requested with the context.

There are two types of context :
1. InitContext - provides context for initialization with input_dimensions
filled. This contains specifications for all the inputs/outputs/weights/tensors
which will be made available during executiong.
2. RunContext - provides allocated weights/inputs/outputs/tensors along with
some of the basic properties of the layer like trainable, etc.

This design allows layer devel to be stateless (derived layer can be stateful with their
own member objects), and behave as layer ops.
Further, this allows dependencies on weights/var_grads as context simplified the interface
with just tensors.
Also, this patch also removes dependency on the manager from layer_devel header.

layer context for each layer is owned by the corresponding LayerNode.
Many of the layer getters like getInputs/Outputs/Weights/etc can now
be moved to LayerNode which does not concern the custom layer writer,
and simplifies the layer operation code as well.

See also #986

**Self evaluation:**
1. Build test: [x]Passed [ ]Failed [ ]Skipped
2. Run test: [x]Passed [ ]Failed [ ]Skipped

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[C++17] Bump base version to c++17

This patch bumps baseline version to c++17.

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[deb/pkg] Updates for debian packaging

Update tolerance for sqrt unittest for debian arm64 packaging
increase timeout for appdraw classification app for debian armhf packaging

**Self evaluation:**
1. Build test: [x]Passed [ ]Failed [ ]Skipped
2. Run test: [x]Passed [ ]Failed [ ]Skipped

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[Props] Implement input_layers as a property

This patch implements input_layers as a property with following changes
below

**Changes proposed in this PR:**
- Add ConnectionSpec object for generic purpose
- Add InputSepc as a property
- Add following tests

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Test/models] Add addition test

This patch adds addition test, while add support for reading multiple
inputs and outputs inside `unittest_nntrainer_models.cpp`

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Retouch] unittest_models test cases

**Changes proposed in this PR:**
- Reduce scale of inputs
- Add iterations for recurrent layers (and delete _n) family tcs
- Clean duplicated codes in genModelTests.py

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[deb/pkg] Increase c-api unittest timeout

Increase c-api unittest timeout even further to
pass armhf build on launchpad.

**Self evaluation:**
1. Build test: [x]Passed [ ]Failed [ ]Skipped
2. Run test: [x]Passed [ ]Failed [ ]Skipped

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[Docs] Update custom documentation for loss layer

This patch updates documentation of custom for loss layer.
Also deletes redundant `pow.cpp` that sneaked in.

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[deb] Update lower tolerance bug + increase timeouts

Inverse the check for lower error tolerance for debian build
Also increase timeouts for nntrainer capi/ccapi unittests
as they timeout for armhf architecture on launchpad.

See also #1251

**Self evaluation:**
1. Build test: [x]Passed [ ]Failed [ ]Skipped
2. Run test: [x]Passed [ ]Failed [ ]Skipped

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[Trivial] Change return -> retval

When stating value in doxygen it is recommended to use @retval, this
patch reflects the issue

**Self evaluation:**
1. Build test: [ ]Passed [ ]Failed [X]Skipped
2. Run test: [ ]Passed [ ]Failed [X]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[interpreter] Updated export_to usage with LayerNode

Updated export_to function usage based on LayerNode

**Self evaluation:**
1. Build test: [x]Passed [ ]Failed [ ]Skipped
2. Run test: [x]Passed [ ]Failed [ ]Skipped

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[layerNode] Refactor get/set number of inputs/outputs

Refactor get/set number of inputs/outputs.
This is now handled by layer_node.
Updating the input/output layers updates the count.

**Self evaluation:**
1. Build test: [x]Passed [ ]Failed [ ]Skipped
2. Run test: [x]Passed [ ]Failed [ ]Skipped

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[layer/node] Move input/output layers to LayerNode

This patch moves input/output layers to LayerNode.
Also create an interface for accessing/setting input/output layers
from in the LayerNode.

**Self evaluation:**
1. Build test: [x]Passed [ ]Failed [ ]Skipped
2. Run test: [x]Passed [ ]Failed [ ]Skipped

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[api] Added api dependency in pc files

Added API dependecy in pc files for the capi and ccapi.
The dependency is of the capi-ml-common which provides ml-api-common.h header.
Further, ml-api-common.h header from nntrainer repo is also removed.
Updated android build to download ml-api-common.h for nnstreamer daily builds.

**Self evaluation:**
1. Build test: [x]Passed [ ]Failed [ ]Skipped
2. Run test: [x]Passed [ ]Failed [ ]Skipped

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[ RNN ] Add Multi-Layer RNN Unit Tests

This commit includes,
 . Multi-Layered RNN Unit Tests
 . Some Fixes to run

**Self evaluation:**
1. Build test:  [X]Passed [ ]Failed [ ]Skipped
2. Run test:  [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: jijoong.moon <jijoong.moon@samsung.com>

[Trivial] Fix NDK warning -Wbraced-scalar-init

This patch fixes trivial NDK warning below

```
././../nntrainer/layers/lstm.cpp:315:25: warning: braces around scalar initializer [-Wbraced-scalar-init]
  Tensor dh_nx = Tensor({derivative_.width()});
                        ^~~~~~~~~~~~~~~~~~~~~
././../nntrainer/layers/lstm.cpp:316:25: warning: braces around scalar initializer [-Wbraced-scalar-init]
  Tensor dc_nx = Tensor({derivative_.width()});
                        ^~~~~~~~~~~~~~~~~~~~~
././../nntrainer/layers/lstm.cpp:352:26: warning: braces around scalar initializer [-Wbraced-scalar-init]
        hs_prev = Tensor({hs_t.width()});
                         ^~~~~~~~~~~~~~
././../nntrainer/layers/lstm.cpp:354:26: warning: braces around scalar initializer [-Wbraced-scalar-init]
        cs_prev = Tensor({cs_t.width()});
```

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[test/model] Cleanup unittest_nntrainer_models

Cleaned up unittest_nntrainer_models.

**Self evaluation:**
1. Build test: [x]Passed [ ]Failed [ ]Skipped
2. Run test: [x]Passed [ ]Failed [ ]Skipped

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[layer] Support inPlace() as an interface for the layer

Support inPlace() as an interface for the layer.
If returns true, manager can set its inputs/outputs to be
the same memory location, but not necessarily.
Corresponding graph code is also updated.

Resolves #867

**Self evaluation:**
1. Build test: [x]Passed [ ]Failed [ ]Skipped
2. Run test: [x]Passed [ ]Failed [ ]Skipped

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[Layer] Add requireLabel() function

Add a simple function `Layer::requireLayer()`.
This function is used to detect if a certain layer requires a layer.

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[APP] Add tensorflow based product rating

Added tensorflow based product rating.

**Self evaluation:**
1. Build test: [x]Passed [ ]Failed [ ]Skipped
2. Run test: [x]Passed [ ]Failed [ ]Skipped

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[SplitLayer] Support flexible dimension

This patch adds support for the flexible dimension in split layer.

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Cc: Parichay Kapoor <pk.kapoor@samsung.com>
Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Permute] Add permute layer forward/backward

This patch implements forward/backward with a following test

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[ RNN ] Add unittest case & return_sequence

This PR includes.
  . unittest of rnn layer
  . support of return sequence of rnn layer

**Self evaluation:**
1. Build test:  [X]Passed [ ]Failed [ ]Skipped
2. Run test:  [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: jijoong.moon <jijoong.moon@samsung.com>

[Recorder] Add multiout equivalent in transLayer

When getting gradient from `gradientTape` it adds all derivatives of the
dependent layer. While this behavior is true, we need to measure to what
extent each layer contributed to the layer.

This patch adds a multiout layer which consists of some stublayer that
seperates derivatives one by one.

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[ LSTM ] Add Multi-Layerd LSTM Unittest

This commit includes,
  . Multi-Layerd LSTM Unit test.

**Self evaluation:**
1. Build test:  [X]Passed [ ]Failed [ ]Skipped
2. Run test:  [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: jijoong.moon <jijoong.moon@samsung.com>

[Permute] Implement peripheral functions

This patch implements initialize, export and setProperty of permute
layer

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[layer] Move distribute to LayerNode

move distribute to LayerNode from layer
Further, the distribute layer is now supposed to be used
as a wrapper of a layer maintained by the LayerNode,
and must not be maintained externally.
Rather, use getDistribute() to check if the node is distributed
and treat the layer as a regular layer.

**Self evaluation:**
1. Build test: [x]Passed [ ]Failed [ ]Skipped
2. Run test: [x]Passed [ ]Failed [ ]Skipped

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[Layer] Add permute layer scaffolding

This patch adds permute layer scaffolding

**Minor Fix**
- Add error handling for transpose tensordim

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[ LSTM ] Add More Unit Test Cases

This Commit inclues,
 . Additional unittest
   - with return sequneces
   - with multiple batch size
   - with multiple batch + return sequences
   - with multiple batch + return sequences + multi-iteration
 . Fixed initialization bug during training
 . Remove check for tensor dot product to support multi-dimension
 tensor.

**Self evaluation:**
1. Build test:  [X]Passed [ ]Failed [ ]Skipped
2. Run test:  [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: jijoong.moon <jijoong.moon@samsung.com>

[ LSTM ] Add Unit test for LSTM Layer

This commit includes,
  . unit test case for lstm layer
  . fixup of potential bug : tanh prime calculation
  .

**Self evaluation:**
1. Build test:  [X]Passed [ ]Failed [ ]Skipped
2. Run test:  [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: jijoong.moon <jijoong.moon@samsung.com>

[ LSTM ] Add return_sequence

This commit includes,
  . impelementation of return_sequence. If it is true, then it
  generate all the output of time iteration and if it false, then
  only last output is available.
  . add 'return_sequence' keyward

Resolves:

**Self evaluation:**
1. Build test:  [X]Passed [ ]Failed [ ]Skipped
2. Run test:  [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: jijoong.moon <jijoong.moon@samsung.com>

[ LSTM ] LSTM Unit Test Cases

This PR includes,
 . LSTM Unittest Case
 . python code in genModels

Resolves:

**Self evaluation:**
1. Build test:  [X]Passed [ ]Failed [ ]Skipped
2. Run test:  [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: jijoong.moon <jijoong.moon@samsung.com>

[ LSTM ] Fix implementation & add backwarding testcase

This commits includes:
  . fix implementation ( apply actvation )
  . add one unittest to check functionality of backwarding

**Self evaluation:**
1. Build test:  [X]Passed [ ]Failed [ ]Skipped
2. Run test:  [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: jijoong.moon <jijoong.moon@samsung.com>

[Exporter] Implement saving fc node

**Changes proposed in this PR:**
- This patch add saving fullyconnected layer with node_exporter
- add dedeicated buffer to `TfNode`

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[app] reduce Logistic regression application epochs

Reduce logistic regression application epochs.
This application is to check working than verify accuracy and can
run with fewer epochs.

**Self evaluation:**
1. Build test: [x]Passed [ ]Failed [ ]Skipped
2. Run test: [x]Passed [ ]Failed [ ]Skipped

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[Recorder] Add Input map generation

There was a bug that layer input is directly from the previous layer.
If the model is not sequential, this assumption is not true, making
Addition test fail.

This patch build `model.recoder__input_maps` to correctly locate
 the input tensor with some minor clean ups

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[meson] Enable tflite-interpreter by default

Enable tflite-interpreter by default as tflite layer is
already enabled.
Move flatc dependency out to the main meson.build
Also add proper path for the generated tf_schema

**Self evaluation:**
1. Build test: [x]Passed [ ]Failed [ ]Skipped
2. Run test: [x]Passed [ ]Failed [ ]Skipped

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[layer] Move flatten property to layerNode

Move flatten property out to layerNode
also prepare for moving other properties out to layerNode

**Self evaluation:**
1. Build test: [x]Passed [ ]Failed [ ]Skipped
2. Run test: [x]Passed [ ]Failed [ ]Skipped

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[Refactor] Delegate tfNode creation to exporter

This patch enables delegating exporting logic to exporter.
Note that this is supposed to be 1:1 mapping, if it has to be m:n
mapping, it should be directly handled by the interpreter, which
actually can see the node.

Also s/`save_result`/`saveResult` and `get_result`/`getResult`

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>
[Android] Add tflite interpreter option

This patch add tflite interpreter option to ndk build

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Fix] Initialize member variable

 - Initialize member variable sorted
 - Catch exception when create the tensor

resolves: 1231330, 1143556

Signed-off-by: hyeonseok lee <hs89.lee@samsung.com>

[deb/pkg] Reduce error tolerance for release build

Reduce error tolerance for release builds to succeed build on launchpad and tizen builds.
The default error tolerance is only reduced for all release builds for all tests that use defined tolerance.
However, the CI will run at original tolerance for all the builds and will ensure that
values are matched with original precision.

The tolerance has to be reduced because different os
have different versions and implementations of the blas
which results in minor difference in the results of the float
operations. However, when the difference is accumulated over
a range of operations, the unittests fail to match the
exact values.

**Self evaluation:**
1. Build test: [x]Passed [ ]Failed [ ]Skipped
2. Run test: [x]Passed [ ]Failed [ ]Skipped

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[Tflite] Separate `tfopnode`

This patch separates tfopnode to a file.

tfOpNode will be directly built from the exporter if it doesn't need to
be fused.

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[tflite] Update dependency from tflite to tf2lite

Update dependency from tensorflow-lite to tensorflow2-lite.
This is done for Tizen and Debian build. It has already been done for Android.

Tensorflow2-lite header is dependent on flatbuffer, so flatbuffers-devel
is additionally added as a build-dependency.

**Self evaluation:**
1. Build test: [x]Passed [ ]Failed [ ]Skipped
2. Run test: [x]Passed [ ]Failed [ ]Skipped

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[pkg/spec] Update spec packaging for tflite layer

Update spec packaging for tflite layer with tflite dependency

**Self evaluation:**
1. Build test: [x]Passed [ ]Failed [ ]Skipped
2. Run test: [x]Passed [ ]Failed [ ]Skipped

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[test] Remove run_unittests

Remove run_unittests

**Self evaluation:**
1. Build test: [x]Passed [ ]Failed [ ]Skipped
2. Run test: [x]Passed [ ]Failed [ ]Skipped

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[deb/pkg] Minor update to deb packaging

Update debian packaging:
- update the unittests to use meson tests
- update dh_missing to fail than give warnings

**Self evaluation:**
1. Build test: [x]Passed [ ]Failed [ ]Skipped
2. Run test: [x]Passed [ ]Failed [ ]Skipped

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[unittest] Update unittest tolerance for debian build

Update unittest tolerance to be 1e-5 for debian build to match
across platforms (bionic, focal) and architectures.
Similar issues have occured for i586 tizen as well.
So, this patch lowers the tolerance for all ccapi and capi test.

This issue occurs to os and library versioning issues.

**Self evaluation:**
1. Build test: [x]Passed [ ]Failed [ ]Skipped
2. Run test: [x]Passed [ ]Failed [ ]Skipped

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[deb/pkg] Add nnstreamer filter and update testing

Add nnstreamer filter in the packaging
Also update the unittests to use the meson tests

**Self evaluation:**
1. Build test: [x]Passed [ ]Failed [ ]Skipped
2. Run test: [x]Passed [ ]Failed [ ]Skipped

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[ Release ] NNTrainer 0.2.0 Release

NNTrainer v0.2.0 is released.

**Self evaluation:**
1. Build test:  [X]Passed [ ]Failed [ ]Skipped
2. Run test:  [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: jijoong.moon <jijoong.moon@samsung.com>

[deb/pkg] Update debian install filenames

Update debian install filenames for the capi and ccapi.

**Self evaluation:**
1. Build test: [x]Passed [ ]Failed [ ]Skipped
2. Run test: [x]Passed [ ]Failed [ ]Skipped

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[Product Ratings] Fix inconsistent batch

This patch fixes inconcistent batch size between source file and model
configuration.

Also found a bug #1238, which will be dealt seperately.

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[debian] Refactor debian packaging

Refactor debian package to make separate packages for capi and ccapi, along with their devels.
Other bug fixes/improvements are:
- Update the names of the packages in pkg-config files for the capi and ccapi
- Update the installed files based on the requirement by the devel headers
(this might be even expanded). This will be refactored again once layer_internal and neuralnet
cleanup is performed.
- Update the included headers

**Self evaluation:**
1. Build test: [x]Passed [ ]Failed [ ]Skipped
2. Run test: [x]Passed [ ]Failed [ ]Skipped

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[APP] Change embeding to product rating

This patch changes embedding application to product rating application
which is more practical use-case

**Changes proposed in this PR:**
- Update Readme
- Change directory and app name embedding->productratings
- tune model and sample dataset

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[tflite] implement build operations in subgraph

This patch implements build operation in the subgraph

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[context/optimizer] Enable plugged optimizer

Enable plugged optimizer with the app_context

**Self evaluation:**
1. Build test: [x]Passed [ ]Failed [ ]Skipped
2. Run test: [x]Passed [ ]Failed [ ]Skipped

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[optimizer] Update optimizer setProperty interface

Update optimizer setProperty interface based on the requirement
from the custom optimizer.

**Self evaluation:**
1. Build test: [x]Passed [ ]Failed [ ]Skipped
2. Run test: [x]Passed [ ]Failed [ ]Skipped

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[Example] Concat example

This patch adds concat example with embedding

The structure is

      (input)
     /       \
  (embed1)   (embed2)
    \         /
      (concat)
         |
      (flatten)
         |
        (fc)

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Fix] Prevent creating multiout for split

This patch make an exception for split layer to not create multiout
layer. While fixing trivial build error of sign compare

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[split/test] Added unittests for split layer

Added unittests for split layer covering initialization,
forwarding and backwarding. Corresponding bug fixes are
also added.
This has revealed an issue with concat layer #1226
which will be resolved soon.

Also add split layer to android build

Resolves #1208

**Self evaluation:**
1. Build test: [x]Passed [ ]Failed [ ]Skipped
2. Run test: [x]Passed [ ]Failed [ ]Skipped

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>