[layer] Performance optimizations to LSTMCell

Add minor performance optimizations to LSTMCell.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[tensor] Add beta for tensor sum

Add beta for tensor to enable addition to existing tensor while summing.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[API/trivial] Add LSTMCell creator

This patch add lstm cell creator

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[layer] override setBatch in bn layer

 - Override setBatch function

Self evaluation:

Build test: [X]Passed [ ]Failed [ ]Skipped
Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: hyeonseok lee <hs89.lee@samsung.com>

[layer] LSTM cell batchresize bug fix

As LSTM cell uses the batch values not in the batch dimension but in the
inner dimension, it creates bug with the existing layer context interface
as it only allows updating the batch along the batch dimension.
This patch provides a quick fix to combine the time and batch values in
the batch dimension.

Once recurrent realizer allows multiple inputs, the time factor will be
removed and the reshapes can be removed.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[layer] Optimization for LSTM cell

Optimization for LSTM cell to combine the activation function running on
the values.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[Android] Add missing fopenmp

This patch add missing fopenmp.

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Android] Enable test for the android build

This patch enable android build, this abling this option has an issue

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Android] Unlink ml-api-inference module

This patch unlink ml-api-inference module as it requires gstreamer

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Fix/pkgconfig] Fix nntrainer include dir

This patch fixes pkg config nntrainer include directory for the pkg.

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[LSTMCELL] Fix compile error

This patch fix compile error in the main branch

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[layer] Bug fix for LSTMCell

Enable recurrent unrolling tests for LSTMCell with a single and stacked
LSTM cell. Add corresponding bug fixes.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[layer] Add first draft of lstm cell

This patch adds the first draft of lstm cell. This has certain
advantages over the existing LSTM when executing for single timestep:
- does not need to divide the data over smaller chunks for computation.
This saves the overhead of creating tensors, an extra loop and combines
computation to a single call for efficiency.
- some of the operations have been avoided based on the current timestep
(which were zeros).
More optimizatios will be coming soon for LSTM cell.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[tensor] Add support for strided operators

This patch adds support for strided multiply, apply and getSharedDataTensor
operations for tesnor.
Later these will be optimized direct data access with an iterator.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[Android] disable nnstreamer layer

This patch disables nnstreamer backbone because of the package issue for
android build

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Clean] remove enable-android option

This patch remove enable-android for brevity

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Android] Add prebuilt android for the built library

This patch provides prebuilt android when making android zip file

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Android] Add simple package script

This patch adds android simple package script to make tar.gz

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Fix] fix bug for install_headers

Fix bug that install_headers install_dir overrides subdir props.
This is undocumented behavior so changed to make it to be deterministic

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Android] Set up install directory for android build

This patch sets up install directory for android build.

It is defaulted to installed in the meson.build_root() /
android_build_result

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Android] Enable capi/ccapi build

This patch enable capi/ccapi build

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Meson] Add android nntrainer build

This patch add android meson build.
which can be triggered by ninja

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Meson] Save and apply extra defines

This patch saves extra defines as a list to be applied for the android
build as well

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Android] disable building src through ninja

This patch disables building src through ninja for android

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[android] Add platform android and install deps

This patch add platform android and install deps according to it

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[TRIVIAL] Fix error log

error log string was concated by '+' which was invalid

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Android] Remove bfd linker

bfd linker is deprecated so it's also deleted from android build

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Trivial] Add override to functions

This patch add override to virtual functions mainly to suppress compile
warnings from android build

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Optimizer] Prevent weight allocation for dependent nodes

This patch prevents optimizer weight allocation for dependent nodes

Resolves #1671

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[build] Add openmp dependency for improved performance

This patch adds openmp dependency while building for improved
performance.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[tensor] Bug fix for dropout mask generation

This patch adds fix for dropout mask generation. It was being
out-of-place which was leading to not using the tensor memory already
allocated. This patch adds the interface to use the already allocated
tensor and use it.
This patch further saves memory while generating dropout mask itself as
well.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[SVACE] Fix init problem

This patch fixes initiation problem

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[mem-opt] Add script to memory peak RSS memory

This patch adds a script to measure the peak RSS memory.
To check the memory consumption, pass the command to run the
applicaiton/test to this script.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[Fix] not used variable

Not used variable is deleted from static analyzer

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Fix] Svace issue

Fix uinit ctor

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Concat] Add axis selection

Add axis selction mechanism if axis is not specified that if height
exist, concat to the width dimension

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Reshape] Add exportTo

This patch implements exportTo to reshape

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Fix] fix validate Input

This patch fixes validate Input

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[test/bn1d] Add bn1d support

This patch adds bn1d support for the transLayer_v2

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Fix] fix recorder registration

This patch fix registration bug when there is more than two overloaded
delgation

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Fix] Layer node clone configuration

layer node clone configuration was not copying realization, this is
fixed with this patch

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[layer] Create property for axis in batch normalization

This patch convert the axis member in batch normalization to a property
to allow setting it externally.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[profile] Support profiling by layer type

This patch adds support for profiling of the model based on the type of
the layers.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[network/neuralnet] Move backwarding to network

The iteration of the graph to perform the backwarding has been moved to
the network graph in this patch.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[layer] Support concat along all the dimensions

This patch provides support for the concat layer among all the
dimensions. Other changes include:
- split_dimension property keyword has been changed to axis and reused
for concat_dimension
- TensorDim::getNumDim() is now a static function

Corresponding unittests for the concat layer are also added.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[Test] Add recurrent value compare tests

This patch adds recurrent value compares golden tests

There are 4 cases presented.

1. single fc recurrent
2. stacked fc recurrent
3. single lstm recurrent
4. stacked lstm recurrent

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[LSTM] Fix single time step backwarding

This patch updates single time step backwarding to properly set and see
the last timestep values

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Test] Add model v2 test generator

This patch add model v2 test generator using pytorch.

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Test] Connect v2 test to the parameterized test

This patch connect v2 test to the model param test for convenience

+ add validateFor_v2. This is separated out because label dimension is
not given, although it should be merged to validateFor

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Test] Add models v2 test

This patch proposes models v2 test which is much lighter.

**Purpose**
1. Lighter model test as layer golden test compensates for it.
2. Easier debugging especially by checking offset, and moving metatdat
3. Give greater flexibility to the golden test generator on the model
architecture
4. Not saving golden file but making it on the fly on the builder side

**Major Difference**
1. Gradient comparison is skipped as it is automatically checked at the
next iteration.
2. Add metadata about num_iterations to file
3. Add size check while reading the golden file
4. Delete redundant information that can be infered.
5. Every iteration has it's own input information.

**Golden Format**
```
    ## file format is as below
    # [<number of iteration(int)> <Iteration> <Iteration>...<Iteration>]
    # Each iteration contains
    # [<input(Tensors)><Label(Tensors)><Parameters(Tensors)><Output(Tensors)>]
    # Each tensor contains
    # [<num_elements(int32)><data_point(float32)>...<data_point(float32)>]
```

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Trivial] Open sizeCheckedReadTensor to util

This patch opens sizeCheckedReadTensor to util for the later use

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Fix/Sharing] Fix gradient updation log

Gradient should be initialized at the very first (backward) access, but the
shared weight was updated at the very last (backward) access, this patch
resolves the issue by adding isFirstAccess to the var_grad.h

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[layer] Support reshape layer

This patch provides support for reshape layer and basic unittests.
The flatten layer is also updated to use reshape layer internally.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[test] GRU with reset_after=False

GRU matches with tf1 but not with tf2.
Tf2 changes the reset_after to True as default, which leads to an extra
bias. This patch forces the reset_after to be always false to enable
matching with tf1 and tf2.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[tensor] Add checks for external tensors

This patch adds checks while setting the external tensors:
- ensure that the external tensor size is more than the required memory
- if the tensor size is 0, its data must be null

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[layer/test] Add unittests for conv1d

This patch adds layers golden tests for conv1d.
conv1d is also added to the list of transpose layers while saving data
from tensorflow.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[layer] Add conv1d implementation

This patch adds conv1d implementation support.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[properties] Bug fix for padding compute

This patch provides bug fix for padding compute in the usage of stride.
The stride along width and height were being used wrongly.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[LSTM] Add lstm fix and test run

This patch add lstm fix and test run.

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Neuralnet] Add debug mode to check optimizer

This patch add debug mode to check optimizer

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Fix] tensor pool prerequest bug

This patch fix tensor pool prereuqestedrequest bug by that shared_name
was wrong

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Trivial] Move makeGraph function out

this patch moves makeGraph function from compiler_util -> test_util

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Test] Make model test to take arbitrary nn

This patch updates model test to accept arbitrary neuralnetwork instead
of ini only

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Trivial] Extract golden test from nnt_models

This patch extracts golden test from nntrainer models, nothing done.

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Trivial] Separate Watchers into a new file

This patch separates watchers into a new file

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[mem-opt] Support in-place batch normalization

This patch provides support for in-place batch normalization.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[batchnorm] Optimize batch norm memory usage

This patch optimized batch norm memory usage by updating the exec order
for its input as its not used in calcDerivative.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[in-place] Support in-place activation

This patch adds support for in-place activation. Further, the in-place
optimization has been moved to the model compile phase.
This patch also adds support for running the model in in-place as well
as out-of-place mode.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[LayerGroups] Normalize identifiers

This patch normalize identifiers inside addWithRefrenceLayers() for further use

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Refactor] Recurrent realizer with in/out props

This patch adds recurrent realizer with in/out props

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Realizer] Implement input realizer

This patch implements input realizer which connect input to external
input

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Realizer] Implement slice realizer

This patch implements slice realizer and it's test

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[layer/test] Bug fix for dropout + unittest

This patch adds bug fix for dropout for training as well as inference
mode. Futher, added unittest:
- when dropout_rate is 0 or 100%, all the values are checked
- when dropout_rate between 0 and 100, weak check ensures that either
values are equal or one of the values (golden vs output) is 0
- when dropout_rate r between 0 and 100, strong check ensures that
100 - 2*r percentage of values must always match

All the checks for performed in the unittests.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[Recurrent] Add timestep property to recurrent layers

This patch creates recurrent layer setting property support to the
recurrent wrapper.

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[packaging] Debian launchpad buildfix for focal

This patch provides buildfix for focal buildfix for launchpad.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[network] Bug fix for trainable with supportBackwarding

This patch resolves the bug for calling calcDerivative on layer which
do not support backwaring.
The issue rises from the assumption that there will be no layer
requiring backwarding after the last non-trainable layer. But this is
not valid for multi-input scenario. With multi-input, there can be over
two input layers where they both do not support backwarding but the rest of
the model can be trainable.

This patch changes this error check. Below are the updated semantics:
1. After initialization of the graph, a check is added to ensure that
for each trainable layer, all the layers all of ahead must support
backwarding, so that the trainable layer can be trained. If any layer
ahead of it does not support backwarding, error is thrown.
2. A layer is only trainable if its trainable property is set to true
(defaults to true) and contains at least 1 weight. If a layer does not
contain any weights, the layer is treated as non-trainable.
3. When backwarding the model, backwarding is called only for layers
which support backwarding, and skipped for layers which donot.

The updated semantics ensure the dependency of the flow of the
derivatives and allows mixture of layers which support and do not
support backwarding.

Resolves #1017

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[Model/API] renew addWithReferenceLayers

This patch add refactored version of addWithReferenceLayers

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Model] Add Model::addWithReferenceLayers

This patch add addWithReferenceLayers prototype

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[layer] Support setBatch for attention layer

This patch provides support for setBatch for attention layer.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[layer] Add constructor for attention layer

This patch adds constructor for attention layer.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[layer] Support single timestep for lstm

This patch adds support for single timestep for lstm.
This is achieved with two external properties:

1. timestep - provides the current timestep for which lstm will run
2. max_timestep - the maximum timestep till which lstm will run

This patch also verifies that this LSTM implementation already does gradient stacking appropriately.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[layer] Bug fix of setBatch for LSTM/GRU/RNN

LSTM/GRU/RNN requested tensors from manager and the shape of tensor
depends on the batch size. However, the layers didnot override the
setBatch to update the batchsize of the request tensors. This patch
provides the corresponding bugfix.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[layer] Remove setBatch for init context

This patch remove setBatch for the init context from the layer
interface. setBatch is now only needed to be set for runContext.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[graph/model] Support for multi-label/input for the model

This patch adds the support for mutli-label while training the model.
- Multi labels/inputs are now allowed for the model by taking the
dimensions from the graph than the first and last nodes
- Outputs are now also taken from the graph for validation

Another bug fix is added related to setBatch. The caches input and label
dimensions were not updated when batchsize was updated in the network
graph. This patch updates the bug for updating the batch size in the
network graph.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[Sharing] SKip saving shared weights

If weight is not original, skip saving it

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Recurrent] Implement finalizing graph

This patch implements finalizing graph with/without return sequence
property

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Recurrent] Implement unrolling

This patch implement unrolling in RecurrentRealizer

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[LayerNode] Add cloneConfiguration function

This patch add cloneConfiguration function, which creates a new node
from an exisiting node

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Recurrent] Add verification and preparation

This patch add logic to verify and add connection from input <->
external inputs

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Realizer] Implement remap realizer

This patch introduce remap realizer which remaps identifier inside a
graph representation. Please refer to the test to see what this realizer
does.

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Recurrent] Add skleton of recurrent realizer

This patch add skeleton of and some basic verification
for the recurrent realizer

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[ Conv1D ] Add Skeleton code for Conv1D Layer

This commit includes:
  . Skeleton code for conv1D
  . Padding1D Property
  . and minor

Signed-off-by: jijoongmoon <jijoong.moon@samsung.com>

[Realizer] Apply flatten realizer

This patch applies flatten realizer to model compile. Later, neuralnet
will not have model_graph until compile() is being called

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Model] Add memory optimization property

This patch add memory optimization property to neuralnetwork. The main
purpose of this is to fixating memory optimzation boolean to be applied
only at neuralnet::compile();

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Test/realizer] Add flatten realizer test

This patch adds flatten realizer test

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Sharing] Implement tensor sharing

This patch implement tensor sharing.

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[test/layers] Add gru layer testing

This patch added gru layer unittest for layer golden tests.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[Graph] Add realizer test skeleton

This patch add realizer test skleton with separating utils to compiler
test utils

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Graph/recurrent] Add concept of realizer

This patch add graph realizer. Graph realizer will preprocess graph
which can be effectively done as a lowering process of compile

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Interpreter] Change signature of interpreter

Instead of returning networkgraph from the interpreter, it returns the
graph representation, which is a specification to generate a graph.

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>