[activation] Derivative for activation

The derivative of softmax has been hand crafted to be different from others
Refer to https://github.com/nnstreamer/nntrainer/blob/2a650512813db6ce3bba828b5790066fbc655f14/nntrainer/src/fc_layer.cpp#L265 for original implementation
Softmax requires softmax(x) as input for derivative while other activations require x as input for derivative_

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>

diff --git a/nntrainer/src/activation_layer.cpp b/nntrainer/src/activation_layer.cpp
index e8fa9ef..8f460d5 100644 (file)
--- a/nntrainer/src/activation_layer.cpp
+++ b/nntrainer/src/activation_layer.cpp
@@ -58,7 +58,10 @@ Tensor ActivationLayer::forwarding(Tensor in, int &status) {
 }
 
 Tensor ActivationLayer::backwarding(Tensor derivative, int iteration) {
-  return derivative.multiply(_act_prime_fn(hidden));
+  if (activation_type == ActiType::ACT_SOFTMAX)
+    return derivative.multiply(_act_prime_fn(hidden));
+  else
+    return derivative.multiply(_act_prime_fn(input));
 }
 
 /**