[docs] Update System Test Document (STD) (#9079)

Added output of NN Compiler and NN Runtime system tests

Signed-off-by: Alexander Efimov <a.efimov@samsung.com>

diff --git a/docs/project/19_NN_Compiler_and_Runtime_STD.rst b/docs/project/19_NN_Compiler_and_Runtime_STD.rst
index 243d7c0..5b99613 100644 (file)
--- a/docs/project/19_NN_Compiler_and_Runtime_STD.rst
+++ b/docs/project/19_NN_Compiler_and_Runtime_STD.rst
@@ -381,6 +381,8 @@ SW System Test Case
 SW System Test Result
 =====================
 
+Scripts for testing coould be found in https://github.sec.samsung.net/p-iliutchenk/nnas_system_tests
+
 Test Schedule
 -------------
 
@@ -411,9 +413,201 @@ Test Schedule
 |            +-----------+---------+-----------+--------+                 |
 |            | Schedule  |   MM    | Schedule  |  MM    |                 |
 +============+===========+=========+===========+========+=================+
-| R.1        |  09/2019  |   1     |           |        |                 |
-+------------+-----------+---------+-----------+--------+-----------------+
-| R.2        |  11/2019  |   1     |           |        |                 |
+| R.1        |  11/2019  |   1     |  12/2019  |  1     |                 |
 +------------+-----------+---------+-----------+--------+-----------------+
-| R.3        |  12/2019  |   1     |           |        |                 |
+| R.2        |  12/2019  |   1     |           |        |                 |
 +------------+-----------+---------+-----------+--------+-----------------+
+
+Test Round 1
+------------
+
+Test Output
+```````````
+
+**NN Compiler tests**
+
+::
+
+ ### STARTED COMPILER TESTS ###
+
+
+ [1/6] STARTED TEST: T1
+ 
+ Executed command: ~/nnfw/build/compiler/tf2tflite/tf2tflite ./inception_v3_info.txt ./inception_v3.pb inception_v3.pb.tflite
+ Read './inception_v3_info.txt'
+ Read './inception_v3_info.txt' - Done
+ Import from './inception_v3.pb'
+ Import from './inception_v3.pb' - Done
+ Export into 'inception_v3.pb.tflite'
+ Export into 'inception_v3.pb.tflite' - Done
+ 
+ ### T1 TEST PASSED ###
+ 
+ 
+ [2/6] STARTED TEST: T2
+ 
+ Executed command: ~/nnfw/build/compiler/tf2tflite/tf2tflite ./mobilenet_v2_info.txt ./mobilenet_v2_1.0_224_frozen.pb mobilenet_v2_1.0_224_frozen.pb.tflite
+ Read './mobilenet_v2_info.txt'
+ Read './mobilenet_v2_info.txt' - Done
+ Import from './mobilenet_v2_1.0_224_frozen.pb'
+ Import from './mobilenet_v2_1.0_224_frozen.pb' - Done
+ Export into 'mobilenet_v2_1.0_224_frozen.pb.tflite'
+ Export into 'mobilenet_v2_1.0_224_frozen.pb.tflite' - Done
+ 
+ ### T2 TEST PASSED ###
+ 
+ 
+ [3/6] STARTED TEST: T3
+ 
+ Executed command: ~/nnfw/build/compiler/onnx2circle/onnx2circle ./inception_v2/model.onnx model.onnx.circle
+ Import from './inception_v2/model.onnx'
+ Import from './inception_v2/model.onnx' - Done
+ Export into 'model.onnx.circle'
+ Export into 'model.onnx.circle' - Done
+ 
+ ### T3 TEST PASSED ###
+ 
+ 
+ [4/6] STARTED TEST: T4
+ 
+ Executed command: ~/nnfw/build/compiler/onnx2circle/onnx2circle ./mobilenetv2-1.0/mobilenetv2-1.0.onnx mobilenetv2-1.0.onnx.circle
+ Import from './mobilenetv2-1.0/mobilenetv2-1.0.onnx'
+ Import from './mobilenetv2-1.0/mobilenetv2-1.0.onnx' - Done
+ Export into 'mobilenetv2-1.0.onnx.circle'
+ Export into 'mobilenetv2-1.0.onnx.circle' - Done
+ 
+ ### T4 TEST PASSED ###
+ 
+ 
+ [5/6] STARTED TEST: T5
+ 
+ WARNING:tensorflow:From /home/aefimov/nnas_system_tests/compiler/T5.py:17: The name tf.InteractiveSession is deprecated. Please use tf.compat.v1.InteractiveSession instead.
+ 
+ 2019-11-20 23:35:02.901137: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3399940000 Hz
+ 2019-11-20 23:35:02.901680: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x3d41c00 executing computations on platform Host. Devices:
+ 2019-11-20 23:35:02.901722: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
+ WARNING:tensorflow:From /home/aefimov/nnas_system_tests/compiler/T5.py:19: The name tf.gfile.GFile is deprecated. Please use tf.io.gfile.GFile instead.
+ 
+ WARNING:tensorflow:From /home/aefimov/nnas_system_tests/compiler/T5.py:20: The name tf.GraphDef is deprecated. Please use tf.compat.v1.GraphDef instead.
+ 
+ WARNING:tensorflow:From /home/aefimov/nnas_system_tests/compiler/T5.py:31: The name tf.global_variables_initializer is deprecated. Please use tf.compat.v1.global_variables_initializer instead.
+ 
+ 2019-11-20 23:35:03.388361: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
+ INFO: Initialized TensorFlow Lite runtime.
+ 
+ ### T5 TEST PASSED ###
+ 
+ 
+ [6/6] STARTED TEST: T6
+ 
+ 
+ ### T6 TEST PASSED ###
+ 
+ 
+ ### [6/6] COMPILER TESTS PASSED ###
+
+
+**NN Runtime tests**
+
+::
+
+ ### STARTED RUNTIME TESTS ###
+ 
+ 
+ [1/7] STARTED TEST: T7
+ 
+ Executed command: /home/aefimov/Product/armv7l-linux.release/out/bin/tflite_loader_test_tool --tflite ./inception_v3.pb.tflite
+ [Execution] Stage start!
+ [Execution] Model is deserialized!
+ [Execution] Input data is defined!
+ [Execution] Graph compiled!
+ [Execution] Done!
+ [Comparison] Stage start!
+ [Comparison] TFLite run done!
+ [Comparison] Max difference: 4.82798e-06
+ [Comparison] Outputs is equal!
+ [Comparison] Done!
+ 
+ ### T7 TEST PASSED ###
+ 
+ 
+ [2/7] STARTED TEST: T8
+ 
+ Executed command: /home/aefimov/Product/armv7l-linux.release/out/bin/tflite_loader_test_tool --tflite ./mobilenet_v2_1.0_224_frozen.pb.tflite
+ [Execution] Stage start!
+ [Execution] Model is deserialized!
+ [Execution] Input data is defined!
+ [Execution] Graph compiled!
+ [Execution] Done!
+ [Comparison] Stage start!
+ [Comparison] TFLite run done!
+ [Comparison] Max difference: 1.19209e-06
+ [Comparison] Outputs is equal!
+ [Comparison] Done!
+ 
+ ### T8 TEST PASSED ###
+ 
+ 
+ [3/7] STARTED TEST: T9
+ 
+ [SYSTEM] Linux
+ [RELEASE] 4.14.87-153
+ 
+ ### T9 TEST PASSED ###
+ 
+ 
+ [4/7] STARTED TEST: T10
+ 
+ [TFLITE]...
+ Executed command: /home/aefimov/Product/armv7l-linux.release/out/bin/tflite_benchmark ./inception_v3.pb.tflite
+ Mean: 1978.9ms
+ 
+ [NEURUN]...
+ Executed command: /home/aefimov/Product/armv7l-linux.release/out/bin/tflite_benchmark ./inception_v3.pb.tflite
+ Mean: 568.4ms
+ 
+ ### T10 TEST PASSED ###
+ 
+ 
+ [5/7] STARTED TEST: T11
+ 
+ Executed command: /home/aefimov/Product/armv7l-linux.release/out/bin/tflite_benchmark ./mobilenet_v2_1.0_224_frozen.pb.tflite
+ 
+ Profile data file found!
+ 
+ ### T11 TEST PASSED ###
+ 
+ 
+ [6/7] STARTED TEST: T12
+ 
+ Executed command several times:
+ /home/aefimov/Product/armv7l-linux.release/out/bin/tflite_benchmark ./one_op.tflite
+ [HEScheduler::schedule] backend for Conv2D is acl_cl. Its eft: 1
+ [HEScheduler::schedule] backend for Conv2D is acl_neon. Its eft: 1
+ [HEScheduler::schedule] backend for Conv2D is cpu. Its eft: 1
+ Maximized backend: acl_cl
+ [HEScheduler::schedule] backend for Conv2D is acl_neon. Its eft: 21830
+ Selected backend: acl_neon
+ Backends not equal. Test passed!
+ 
+ ### T12 TEST PASSED ###
+ 
+ 
+ [7/7] STARTED TEST: T13
+ 
+ Executed command:
+ /home/aefimov/Product/armv7l-linux.release/out/bin/tflite_benchmark ./inception_v3.pb.tflite
+ Mean time without scheduler: 569.5ms
+ PROFILING...
+ Mean time with scheduler: 492.5ms
+ 
+ ### T13 TEST PASSED ###
+ 
+ 
+ ### [7/7] RUNTIME TESTS PASSED ###
+
+Defect rates
+````````````
+
+No defects are found
+