From: HyoungJoo Ahn Date: Thu, 17 Oct 2019 12:17:57 +0000 (+0900) Subject: [CAPI/DOCS] upload the first version of guide doc X-Git-Tag: accepted/tizen/unified/20191122.122422~39 X-Git-Url: http://review.tizen.org/git/?a=commitdiff_plain;h=896d35a87157f5768b02d6f48f5be5f3c1735478;p=platform%2Fupstream%2Fnnstreamer.git [CAPI/DOCS] upload the first version of guide doc this is the first version of guide doc Signed-off-by: HyoungJoo Ahn --- diff --git a/Documentation/media/input-selector.png b/Documentation/media/input-selector.png new file mode 100644 index 0000000..d8e39a3 Binary files /dev/null and b/Documentation/media/input-selector.png differ diff --git a/Documentation/media/output-selector.png b/Documentation/media/output-selector.png new file mode 100644 index 0000000..80b7a4d Binary files /dev/null and b/Documentation/media/output-selector.png differ diff --git a/Documentation/nnstreamer_capi.md b/Documentation/nnstreamer_capi.md new file mode 100644 index 0000000..9b86d05 --- /dev/null +++ b/Documentation/nnstreamer_capi.md @@ -0,0 +1,356 @@ +# Machine Learning Inference + + +You can easily create and efficiently execute data stream pipelines that consist of neural networks as filters in pipelines. + +The main features of the Machine Learning Inference API include: + +- Construction of data pipeline based on [GStreamer](https://gstreamer.freedesktop.org/) + + You can compose the data stream pipeline through Machine Learning Inference with various elements of GStreamer and NNStreamer. + +- [Single](#single-api) API and [Pipeline](#pipeline-api) API + + There are two types of Machine Learning Inference API - Single API and Pipeline API. + + Single API is useful for a simple usage scenario of neural network models. It allows invoking a neural network model with a single instance of input data for the model directly. It is useful if you have the input data pre-processed with the application itself and there are no complex interactions between neural network models, data processors, or data stream paths. + + Pipeline API allows developers to construct and execute pipelines with multiple neural network models, multiple inputs and output nodes, multiple data processors, pre-and-post processors, and various data path manipulators. Besides, if the input is online data or streamed data, Pipeline API simplifies your application and improves its performance. + + +- Support various neural network frameworks (NNFW) + + TensorFlow, TensorFlow-Lite, Caffe2, and PyTorch are the supported neural network frameworks. Neural network model files trained by such frameworks can be imported as filters of pipelines directly. + Custom filters, which are neural network models implemented directly with programming languages including C/C++ and Python, maybe imported as filters of pipelines directly as well. + + > **Note** + > + > The devices powered by Tizen OS can contain TensorFlow-Lite only. Ensure that the neural network frameworks that you want to use are installed. + +## Prerequisites + +To enable your application to use the machine learning functionality: + +1. To use the functions and data types of the Machine Learning Inference API, include the `` header file in your application: + + ```c + #include + ``` + +2. To use the Machine Learning Inference API, include the following features in your `tizen-manifest.xml` file: + + ```xml + true + true + ``` + +## Single API + +This section shows how to load a model without the construction of pipelines. + +1. Open a model file: + + ```c + #include + + ml_single_h single; + ml_tensors_info_h in_info, out_info; + + ... + + ml_single_open (&single, "model_file_path", in_info, out_info, ML_NNFW_TYPE_TENSORFLOW_LITE, ML_NNFW_HW_ANY); + ``` + + To load a model file, two `ml_tensors_info_h` are required. `in_info` contains the information of the input tensors, and `out_info` contains the information of the output tensors. For more information, see [Tensors Information](#tensors-information). + +2. Get the [Tensors Information](#tensors-information). + + After opening the model, use the following functions to bring the information of the input and output tensors: + ```c + ml_single_get_input_info (single, &in_info); + ml_single_get_output_info (single, &out_info); + ``` + +3. Invoke the model with input and output [Tensors Data](#tensors-data). + + The model can be invoked with input and output tensors data. The result is included in the output tensors data: + ```c + ml_tensors_data_create (in_info, &input); + ml_single_invoke (single, input, &output); + ``` + +4. close the opened handle: + + ```c + ml_single_close (single); + ``` + +## Pipeline API + +This section shows how to create a pipeline. + +### Basic Usage + +1. Construct a pipeline with the GStreamer elements. + + Different pipelines can be constructed using various GStreamer elements: + ```c + char pipeline[] = "videotestsrc num_buffers=2 ! videoconvert ! videoscale ! video/x-raw,format=RGBx,width=224,height=224 ! tensor_converter ! fakesink"; + ml_pipeline_h handle; + int status = ml_pipeline_construct (pipeline, NULL, NULL, &handle); + ``` + +2. Start the pipeline and get state: + + ```c + /* The pipeline could be started when the state is paused */ + status = ml_pipeline_start (handle); + status = ml_pipeline_get_state (handle, &state); + ``` + +3. Stop the pipeline and get state: + + ```c + status = ml_pipeline_stop (handle); + status = ml_pipeline_get_state (handle, &state); + ``` + +4. Destroy the pipeline. + + When no longer needed, destroy the pipeline: + ```c + status = ml_pipeline_destroy (handle); + ``` + +### Element API + +You need to manipulate the input and the output data to run neural network models with Machine Learning Inference API. In addition, you can construct pipelines that can be controlled. + +Followings are the available elements: + +- **Source** + + The configuration of the data source element is required to set the input tensor data: + ```c + char pipeline[] = "appsrc name=srcx ! other/tensor,dimension=(string)4:1:1:1,type=(string)uint8,framerate=(fraction)0/1 ! tensor_sink"; + ``` + `ml_pipeline_src_get_handle()` controls the `appsrc` element with the name `srcx`: + + ```c + ml_pipeline_h handle; + ml_pipeline_src_h srchandle; + + status = ml_pipeline_construct (pipeline, NULL, NULL, &handle); + status = ml_pipeline_start (handle); + status = ml_pipeline_src_get_handle (handle, "srcx", &srchandle); + ``` + + You can check the information of input tensors using `srchandle`: + + ```c + ml_tensors_info_h info; + + status = ml_pipeline_src_get_tensors_info (srchandle, &info); + ``` + + The input tensor data can be filled according to the `info`: + ```c + ml_tensors_data_h data; + + status = ml_tensors_data_create (info, &data); + + for (i = 0; i < 10; i++) { + uintarray1[i] = (uint8_t *) malloc (4); + uintarray1[i][0] = i + 4; + uintarray1[i][1] = i + 1; + uintarray1[i][2] = i + 3; + uintarray1[i][3] = i + 2; + } + + status = ml_tensors_data_set_tensor_data (data, 0, uintarray1[0], 4); + + /* Setting the policy of raw data pointer */ + status = ml_pipeline_src_input_data (srchandle, data, ML_PIPELINE_BUF_POLICY_DO_NOT_FREE); + ``` + + After using the data source element, release the handle: + + ```c + status = ml_pipeline_src_release_handle (srchandle); + ``` + +- **Sink** + + The configuration of the data sink element is required to get the output tensor data: + ```c + char pipeline[] = "videotestsrc num-buffers=3 ! videoconvert ! tensor_converter ! appsink name=sinkx"; + ``` + + `appsink` element with the name `sinkx` becomes reachable through `ml_pipeline_sink_register()`: + ```c + int status; + ml_pipeline_h handle; + ml_pipeline_sink_h sinkhandle; + + status = ml_pipeline_sink_register (handle, "sinkx", sink_callback, user_data, &sinkhandle); + ``` + + You can get the data from `sink_callback()`, whenever `appsink` named `sinkx` receives data: + ```c + typedef void (*ml_pipeline_sink_cb) (const ml_tensors_data_h data, const ml_tensors_info_h info, void *user_data); + ``` + + Release the `sinkhandle` through `ml_pipeline_sink_unregister()`: + ```c + status = ml_pipeline_sink_unregister (sinkhandle); + ``` + +- **Valve** + + This element is used to control the stream of a pipeline: + ```c + char pipeline[] = "videotestsrc is-live=true ! videoconvert ! videoscale ! video/x-raw,format=RGBx,width=16,height=16,framerate=10/1 ! tensor_converter ! valve name=valve1 ! fakesink"; + int status = ml_pipeline_construct (pipeline, NULL, NULL, &handle); + ``` + + By default, valve named `valve1` of the pipeline is opened. You can control the valve using `ml_pipeline_valve_h`: + ```c + ml_pipeline_h handle; + ml_pipeline_valve_h valve1; + + status = ml_pipeline_valve_get_handle (handle, "valve1", &valve1); + ``` + + After you start a pipeline, you can control the stream of the pipeline with a valve: + ```c + status = ml_pipeline_start (handle); + + status = ml_pipeline_valve_set_open (valve1, false); /* Close */ + ``` + + You can also open the pipeline by controlling the stream of a pipeline with a valve: + ```c + status = ml_pipeline_valve_set_open (valve1, true); /* Open */ + ``` + + Before you destroy the pipeline, release `ml_pipeline_valve_h`: + ```c + status = ml_pipeline_valve_release_handle (valve1); /* Release valve handle */ + ``` + +- **Switch** + + The switch element is used when you need only one working branch from a pipeline that has multiple branches: + + ![input-selector](./media/input-selector.png) + + ```c + char pipeline[] = "input-selector name=ins ! tensor_converter ! tensor_sink name=sinkx videotestsrc is-live=true ! videoconvert ! ins.sink_0 videotestsrc num-buffers=3 is-live=true ! videoconvert ! ins.sink_1"; + ``` + + Get `ml_pipeline_switch_h`. The name of the switch in this pipeline is `ins`: + ```c + ml_pipeline_h handle; + ml_pipeline_switch_h switchhandle; + ml_pipeline_switch_e type; + + status = ml_pipeline_construct (pipeline, NULL, NULL, &handle); + status = ml_pipeline_switch_get_handle (handle, "ins", &type, &switchhandle); + ``` + + You can control the switch using the handle `ml_pipeline_switch_h`: + ```c + status = ml_pipeline_switch_select (switchhandle, "sink_1"); + ``` + + Before you destroy the pipeline, release `ml_pipeline_switch_h`: + ```c + status = ml_pipeline_switch_release_handle (switchhandle); + ``` + + The following image shows the switch at the end of the pipeline: + + ![output-selector](./media/output-selector.png) + + ```c + char pipeline[] = "videotestsrc is-live=true ! videoconvert ! tensor_converter ! output-selector name=outs outs.src_0 ! tensor_sink name=sink0 async=false outs.src_1 ! tensor_sink name=sink1 async=false" + ``` + +### Pipeline States + +For more information about the pipeline states, see [GStreamer guide](https://gstreamer.freedesktop.org/documentation/plugin-development/basics/states.html). + +## Tensors Information + +`ml_tensors_info_h` contains the information of tensors. The tensor info can be managed using the following functions: + +- **Create and destroy** + ```c + ml_tensors_info_h info; + status = ml_tensors_info_create (&info); + status = ml_tensors_info_destroy (info); + ``` + +- **Set functions** + ```c + /* Set how many tensors exist */ + status = ml_tensors_info_set_count (info, 1); + + /* Set the type of the tensor_0 as UINT8 */ + status = ml_tensors_info_set_tensor_type (info, 0, ML_TENSOR_TYPE_UINT8); + + /* Set the dimension of the tensor_0 as in_dim */ + status = ml_tensors_info_set_tensor_dimension (info, 0, in_dim); + + /* Set the name of the tensor_0 as "tensor-name-test" */ + status = ml_tensors_info_set_tensor_name (info, 0, "tensor-name-test"); + ``` + +- **Get functions** + ```c + /* Get how many tensors exist */ + status = ml_tensors_info_get_count (info, &num); + + /* Get the type of the tensor_0 */ + status = ml_tensors_info_get_tensor_type (info, 0, &out_type); + + /* Get the dimension of the tensor_0 */ + status = ml_tensors_info_get_tensor_dimension (info, 0, in_dim); + + /* Get the name of the tensor_0 */ + status = ml_tensors_info_get_tensor_name (info, 0, &out_name); + + /* Get the size of the tensor_0 */ + status = ml_tensors_info_get_tensor_size (info, 0, &data_size); + ``` + +## Tensors Data + +`ml_tensors_data_h` contains the raw data of tensors. The tensor data can be managed using the following functions: + +- **Create and destroy** + ```c + ml_tensors_data_h data; + ml_tensors_info_h info; + + status = ml_tensors_data_create (info, &data); + status = ml_tensors_data_destroy (data); + ``` + +- **Get and set tensor data** + ```c + /* Get tensor data */ + void *data_ptr; + size_t data_size; + status = ml_tensors_data_get_tensor_data (data, 0, &data_ptr, &data_size); + + /* Set tensor data */ + uint8_t dummy[4] = {1, 1, 1, 1}; + status = ml_tensors_data_set_tensor_data (data, 0, dummy, 1); + ``` + +## Related Information + +- Dependencies + - Tizen 5.5 and Higher for Mobile + - Tizen 5.5 and Higher for Wearable