[gradclip] hot fix + unittests
authorParichay Kapoor <pk.kapoor@samsung.com>
Thu, 25 Nov 2021 12:07:57 +0000 (21:07 +0900)
committerJijoong Moon <jijoong.moon@samsung.com>
Wed, 1 Dec 2021 09:54:01 +0000 (18:54 +0900)
commit27907c81bb6ae1936537c24b2e31a9e2b0bfccf9
treed4bb2d95a6cb36771d5ae499644885d2eaa491a7
parent7a9234a225249bc5623709be092e8b4dccea86b6
[gradclip] hot fix + unittests

gradient clipping works by adding extending the execution order of the
gradients to the last node, where the global norm is calculated and the
gradients are clipped and applied.
However, weight sharing use of gradients also relies on the last access
of the gradient and gradient clipping disturbs the balance of gradient
last access.
As a quick fix, if gradient clip is enabled, the last access is replaced
with second to last access.

A better way would be for clipping to be layer, and then last access by
clipping layer would be a valid access and balance to the system can be
maintained.

Unittests for gradient clipping is added with and without weight
sharing.

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>
nntrainer/graph/network_graph.cpp
nntrainer/tensor/manager.cpp
nntrainer/tensor/manager.h
nntrainer/tensor/weight.h
packaging/unittest_models_v2.tar.gz
test/input_gen/genModelsRecurrent_v2.py
test/input_gen/recorder_v2.py
test/unittest/models/unittest_models_recurrent.cpp
test/unittest/unittest_nntrainer_models.cpp