Revert "Load Optimizer Variables"

This reverts commit c669732b1f52f4aad3114839fe1ebba0f5d95f27.

As this commit contains some compatibility breaking changes, this commit
should be merged with where nntrainer is being used.

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

Load Optimizer Variables

In this PR,
 1. new property of adam optimizer, "laod_var" is added to set loading
    momentum variables
 2. update the read and save binary file
    using  ml::train::OptimizerType
 3. update read in layer_node to skip the optimizer variables if
    load_var is set as "false"
 4. is_laod_var() is added in optimizer_devel

**Self evaluation:**
1. Build test:  [X]Passed [ ]Failed [ ]Skipped
2. Run test:  [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: jijoong.moon <jijoong.moon@samsung.com>

[ BUILD ] add lr_scheduler.h in include dir

For the dependency of apt_context.h, we need to install lr_scheduler.h
in install include directory.

**Self evaluation:**
1. Build test:  [X]Passed [ ]Failed [ ]Skipped
2. Run test:  [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: jijoong.moon <jijoong.moon@samsung.com>

[layer] LSTM weight array size fix

this patch adds LSTM weight array size fix.
The lstm layer requests 17 tensors to the tensor manager. However, lstm
layer only maintains an array of size 15 to store the index of the requested
tensors. This leads to an out of range access error.

This patch increases the size of the array to store the requested
tensors indices from 15 to 17.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[capi/test] Example of batchsize set at train

Update capi test to include an example where batchsize is set at the
training time and not at compile time.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[test] Add finalize tests for lr schedulers

Add tests for finalize member function for the learning rate schedulers.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[test] learning rate scheduler unittest

Add unittests for the constant and exponential learning rate schedulers.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[appcontext] Register learning rate scheduler with app context

register learning rate scheduler with the app context and regiter its
type factories.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[spec] Add gcov package

 - Added gcov package for automatic line coverage

Signed-off-by: hyeonseok lee <hs89.lee@samsung.com>

[weight] weight regularization and decay with clip

Enable weighte regularization and decay with weiht clipping.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[weight-decay] Bug fix for weight decay with adam

weight decay should be applied before calling the optimizer
this was not detected earlier as it was tested with sgd

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[optim] Add pytorch reference for adam

Add pytorch reference mode for adam which references adam implementation
of pytorch.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[Load/Save] Update to load and save the adam momentum variables

In this PR, Load and save is updated for the adam optimizer variables.
In order to do it, optimizer load twice, one for the weight, and one
for the optimizer variables.

**Changes proposed in this PR:**
- Added TOC generator for README.md

Resolves:

**Self evaluation:**
1. Build test:  [X]Passed [ ]Failed [ ]Skipped
2. Run test:  [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: jijoong.moon <jijoong.moon@samsung.com>

[lr-scheduler] Add finalize to interface

Add finalize to the interface of the learning rate scheduler with the
purpose to verify that the required properties have been set.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[lr] Support exponential learning rate scheduler

Support exponential decay learning rate scheduler with decay rate and
decay steps are the controllable properties.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[lr] Support constant learning rate scheduler

Support constant learning rate scheduler.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[lr] Add interface for learning rate scheduler

Add interface for the learning rate scheduler which all learning rate
schedulers must abide by.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[optimizer] prepare for learning rate scheduler

Move declaration and definition of learning rate related properties out
to common properties from optimizer_impl class.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[api] Update model inference API

Update model inference to accept inputs and labels which are const
vectors.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[weight-decay] Enable for batch normalization

Enable weight decay for batch normalization.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[test] unittest for weight decay

added unittest for weight decay along with fix.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[layers] Support for weight decay to layers

Add support for weight decay to the layers with weights.
Further update the requestWeight API of the layer context to accept
weight decay, and update the usage at manager.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[weight] Support weight decay

Add support for weight decay property which will enable decay of weights
with each applying of the gradient.
Weight decay can be enabled individually for both weight and bias.
This is kept separate from regularizer as they both behave differently.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[lstm] implement bidirectional lstm forward

 - Make batch_first_forward function
 - For now only support forward for bidirectional lstm

Signed-off-by: hyeonseok lee <hs89.lee@samsung.com>

[ SAVE/LOAD ] save / load optimizer variables

Enable save / load optimizer variables such as M and V for adam
optimizer

**Self evaluation:**
1. Build test:  [X]Passed [ ]Failed [ ]Skipped
2. Run test:  [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: jijoong.moon <jijoong.moon@samsung.com>

[lstm] remove timestep property

 - Remove timestep property from lstm layer.
   This will disable unrolling the lstm layer.
 - Adjust recurrent unittest to simple lstm unittest.

Signed-off-by: hyeonseok lee <hs89.lee@samsung.com>

[Fix] Flatten realizer to maintain original name

This patch changes flatten realizer to maintain original node name.
Please see the included test patch to get the concept.

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Tensor Pool] Add expose/persist concept

This patch add expose/persist concept to the tensor pool and manager.

When a tensor is exposed, this means that the tensor is gauranteed to
remain valid max_exec where max_exec is the value passed along allocateTensors(max_exec);

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[ExecOrder] Add exec order control and fix inference

This patch add exec order control and reduce memory overhead of saving
inference result

exec order cotrol has two part

1. Additional exec order. This enable graph to add some exec order.
2. Expose bit. This enables request to be visible after the end of the
execution.

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Network Graph] Split max_exec_order

This patch split max_exec_order to three part.

1. graph_exec_end: graph execution end
2. backward_iter_end: end node of backward operation
3. forward_iter_end: end node of the forward operation

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[manager] Separate backwarding -> grad, deriv

This patch separate backwaring to grad and deriv for the requestTensors

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Debug] Add a naive validator to the optimized planner

This patch add a naive validator to the optimized planner. This doubles
the memory consumption. So kept commented but when time comes, this
function can be enabled.

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Fix] fix lifespan of recurrent cells

This patch fixes lifespan of recurrent cells

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Fix] change max order to point last

This patch fixes bug that gradient clipping was not applied correctly
when optimization is on.

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Fix] BN life span

t_reduced is used in forward, but lifespan was in backwarding. This
patch fixes the issue

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[compat] remove contrib from tf headers

This patch remove contrib path from tflite headers as contrib headers is
being removed.

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Fix] Non-const reference

This patch fixes returning reference of local variable.

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Android] Delegate option control to android.mk

This patch delegates option control in android mk to meson for debug and
optimized build

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[ Network Graph ] Temporal Fix in getInputGrad

When we turn on the inPlaceOptimization, it will fail if we have
mulitiple inputs: input layer ( can be inplace operatation ), normal
layers. Then, we are not allocate the grad between input layer and
this layer and it cause the error during calcDeriviate which requires
the grad tensor of input layer.

This patch fix this temporally by creating tensor buffer it reqruies.

and it includes modification of mem_check script to generate proper
output.

**Self evaluation:**
1. Build test:  [X]Passed [ ]Failed [ ]Skipped
2. Run test:  [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: jijoong.moon <jijoong.moon@samsung.com>

[grucell] enable multi inout

 - Enable multi inout for grucell
 - Generate grucell layer/model unittest

Self evaluation:

Build test: [X]Passed [ ]Failed [ ]Skipped
Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: hyeonseok lee <hs89.lee@samsung.com>

[grucell] refactoring grucell

 - Rename grucell variables
 - Uncommented genModelTests

Self evaluation:

Build test: [X]Passed [ ]Failed [ ]Skipped
Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: hyeonseok lee <hs89.lee@samsung.com>

[Dropout] Disable test strong match

As dropout is statistcally matching strong output of 60%, it turned out that
it is true only sometime, diabling dropout strong match

Signed-off-by: hyeonseok lee <hs89.lee@samsung.com>

[Recurrent] Add input seqeuncing mechanism

As some kind of recurrent model requires input layer it self to be
changed along the sequence, this patch proposes a simple mechanism to
add time stamp suffix for the input layers as well

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[lstmcell] support multi in/out

 - Refactoring lstmcell layer to support multi in/out (3 input / 2 output)
 - Regenerate lstmcell testcase for multi in/out

Self evaluation:

Build test: [X]Passed [ ]Failed [ ]Skipped
Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: hyeonseok lee <hs89.lee@samsung.com>

[bugfix] zoneout lstmcell regenerate mask

 - Current implementation regenetate the zoneout mask in calcGradient.
    Fix it to reuse the zoneout mask and remove renerating the mask.

Self evaluation:

Build test: [X]Passed [ ]Failed [ ]Skipped
Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: hyeonseok lee <hs89.lee@samsung.com>

[zoneout lstmcell] support multi in/out

 - Refactoring zoneout lstmcell layer to support multi in/out (3 input / 2output)
 - Regenerate zoneout testcase for multi in/out

Self evaluation:

Build test: [X]Passed [ ]Failed [ ]Skipped
Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: hyeonseok lee <hs89.lee@samsung.com>

[Update] Multiout model golden test

This patch generate multiout model golden test

Signed-off-by: hyeonseok lee <hs89.lee@samsung.com>

[Output] Return zero grad for empty output

Along this patchline implements returning zero grad for empty output

Particualary in this patch...

 - Remove manager::requestOutput
 - Use requestTensors instead of requestOutput to request outputs

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Clean] Remove getOutputDimensions()

This patch removes initContext::getOutputDimensions().

This function is only used in the test so removed
(It is used in network_graph but soon will be substitued)

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[context] Receive out conn info and use

Init Context receive out conn info (true if conn info exist, false if
not) and use it to determine given output is dangled

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Manager] update requestInputs to use requestTensor

This patch update requestInputs to use requestTensor_ as a bridge.

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Context] out_dim -> out spec

This patch prepare migration to tensor spec v2 by substituting out
dimension to out specification inside layer_context

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Tensor] OutputGrad defaults to be zero if not given

This patch creates output grad(incoming) to all zero if the given output
is not trainable. This will make the given output be considered as
constant. If a user wants to check if output is constant-like(having
zero gradient as it's partner), she can easily check if
outputHasGradient

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Const] Make incoming derivative const

As incoming derivative shall not be changed by the layer, this patch
changes runcontext::getIncomingDerivatives and runcontext::getOutputGrad
to return const.

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Test] Add test with dangling output

This patch add test with dangling output, this test passes test but
fails optimized run.

Current aim is to make this pass optimized run as well.

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[lstm] refactoring lstm layer

 - Refactoring lstm layer to use lstm cell core functions.

Self evaluation:

Build test: [X]Passed [ ]Failed [ ]Skipped
Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: hyeonseok lee <hs89.lee@samsung.com>

[zoneout lstmcell] refactoring zoneout lstmcell layer

 - Refactoring zoneout lstmcell layer to use lstmcore functions.
 - Preserve lstm_cell_state tensor for calcGradient.
 - Remove lstmcell core layer

Self evaluation:

Build test: [X]Passed [ ]Failed [ ]Skipped
Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: hyeonseok lee <hs89.lee@samsung.com>

[lstmcell] refactoring lstmcell layer

 - Refactoring LSTMCell layer to use LSTM core functions.

Self evaluation:

Build test: [X]Passed [ ]Failed [ ]Skipped
Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: hyeonseok lee <hs89.lee@samsung.com>

[lstmcell core] prepare refactoring lstmcell core layer

 - LSTM cell core layer will be refactoring from layer based to function based.
   This commit prepare the core functions which is not used right now.

Self evaluation:

Build test: [X]Passed [ ]Failed [ ]Skipped
Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: hyeonseok lee <hs89.lee@samsung.com>

[Recurrent] Support connection for as_sequence

This patch enables recurrent realizer to consume support connection
residing in as_sequence parameter

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[RecurrentRealizer] Modify to have connection

This patch modifies recurrent realizer to have connetion

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Realizer] Change input connection semantics

Input connection semantic changed to be more intuitive by mapping
connection one to one

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Realizer] Change slice realizer to get connection

This patch change slice realizer to get connection

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Semantics] InitContext NumOutput is hint

This patch changes init context num output to be a hint how many outputs
has to be allocated. Layers may or may not depend on this information.
What Layer must gaurantee is that number of acutal outputs must be
bigger than the init context num output requested, this fixes dangling,
unused variable issue

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Spec] Change nntrainer install dir

This patch change nntrainer install dir to /usr/prefix/*,
see also https://github.com/nnstreamer/nnstreamer/issues/3560

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Identity] Add identity layer test

This patch add identity layer test

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Identity] Implement and connect identity layer

THis patch implement and connect identity layer

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Identity] Add identity header

This patch add identity layer header.

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Graph] set trainable only if trainable

This patch fixes bug that needsCalcGradient is set true when it
shouldn't be

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[layer] MAE layer bug fix

MAE layer bug fix:
- forwarding fixed to propagate input unchanged
- backwarding updated to scale with size of the tensor

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[optimizer] no request vars for non-trainable weights

optimizer is requesting variables for non-trainable weights, but it
should not.
This patch provides the corresponding fix.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[tensor] Tensor pool request bug fix

If the max execution order for the tensor pool was less than the
largest execution order for a tensor in the tensor pool, then the max
execution order was being included in the execution order of that
tensor.
This patch resolves this issue for end order, as well as for the start
order.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[Tensor] Add tensor cat

This patch add tensor cat method and it's test

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[tensor] Add tensor::split

This patch add tensor split for wider use

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Trivial] Tensor save changes /ofstream/ostream

There is no reason to limit saving to ofstream. Thus it is changed to
ostream

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[API] Add layer visitor for the model

This patch add visitor for the model api

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[gru] enable bias_hh, reset_after

 - Enable bias_hh in gru.
 - Enable reset_after in gru, grucell. If reset_after is set to true,
   apply reset gate after matrix multiplication.

close #1768

Self evaluation:

Build test: [X]Passed [ ]Failed [ ]Skipped
Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: hyeonseok lee <hs89.lee@samsung.com>

[loss] optimize mse part 3

optimize mse layer to use scaled l2norm instead of manually calculating
the l2norm.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[loss] optimize mse part 2

optimize mse backwarding calculation by merging multiple and divide into
a single call.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[loss] optimize mse part1

reduce memory usage for mse forwarding.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[grucell] enable 2 bias

 - Enable bias_hh in grucell

Self evaluation:

Build test: [X]Passed [ ]Failed [ ]Skipped
Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: hyeonseok lee <hs89.lee@samsung.com>

[lstm, lstmcell, zoneout] enable 2 bias

 - Enable bias_hh in lstm, lstmcell, zoneout lstmcell layers

Self evaluation:

Build test: [X]Passed [ ]Failed [ ]Skipped
Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: hyeonseok lee <hs89.lee@samsung.com>

[rnn] enable 2 bias

 - Enable bias_hh in rnn

Self evaluation:

Build test: [X]Passed [ ]Failed [ ]Skipped
Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: hyeonseok lee <hs89.lee@samsung.com>

[rnncell] enable 2 bias

 - Make a integrate bias property. It decide whether integrate 2 bias to 1
   or not. It will be used in rnn variant for now.
 - Added bias_hh in rnncell

Self evaluation:

Build test: [X]Passed [ ]Failed [ ]Skipped
Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: hyeonseok lee <hs89.lee@samsung.com>

[test] enable zoneout lstm test

enable zoneout lstm test

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[layer] bug fix for array initializer

Bug fix for array initializer storing the indices of the requested
weight so that accessing weights not requested always causes error.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[Connection] Enhance parsing logic

This patch enhances parsing logic of connection especially related to
indexing

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Cc: Parichay Kapoor <pk.kapoor@samsung.com>
Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[Trivial] Complement connection doxygen

Add missing parameter explanation in doxygen

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[node/trivial] Change input layers to input conns

This patch chagne input layers local variable name to input connections to
make it consistent

**Self evaluation:**
1. Build test: [X]Passed [ ]Failed [ ]Skipped
2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>

[manager] Remove activation input exec order from backwarding

This patch removes activation input exec order from backwarding as input
of the activation layer is not used in the backwarding.

This leads to change in the unittests as we cannot check all the
outputs, especially close to the end of the model. So, with optimization
enabled, only the output layer's forwarding is checked.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[layer] Disable bias enabled for fc and conv

This patch enables disable bias for fully connected and convolution layer.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[layer] Add disable bias property for layer impl

Add disable bias property for the layer impl class.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[test] Disable zoneout lstm unittests

Disable zoneout lstms unittests till they are fixed.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[test] recoder to read from input from file

support recorder to read data from file than always generating random
data.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[model] Add clip gradient norm property

Add clip gradient norm property for the the model as a simple interface
to set for all the layers.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[layer] Embedding fix

Undo earlier changes due to bug in them.
The changes will be proposed again with more unittests.
The changes have been commented for now.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[layer] zoneout lstm bug fix

zoneout lstm bug fix for handling the scenario where mask rate is set to
zero for either hidden or cell.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[realizer] bug fix for activation realizer

Bug fix for activaiton realizer to find the realizer where the layer is
looping to itself with name change.
TODO: also do for flatten and other realizers which modified the graph
in similar fashion.

Signed-off-by: Jihoon Lee <jhoon.it.lee@samsung.com>
Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[layer] Fixes for mol attention layer

- gradient accumulation
- support only 1 output in case state update is not used
- bug fix in backwarding
- activation to work out of place

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

[realizer] recurrent realizer bug fix in concat

concat to not loop over all inputs multiple times.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>