# Python* Calibration Tool
-The Python* Calibration Tool calibrates a given FP32 model so that you can run calibrated model in low-precision 8-bit integer mode while keeping the input data of this model in the original precision.
-The Calibration Tool is a Python\* command-line tool, which imports Python types from the `openvino.tools.calibration` package.
+## Introduction
+
+The Calibration Tool quantizes a given FP16 or FP32 model and produces a low-precision 8-bit integer (INT8) model with keeping model inputs in the original precision. To learn more about benefits of inference in INT8 precision, refer to [Using Low-Precision 8-bit Integer Inference](./docs/IE_DG/Int8Inference.md).
> **NOTE**: INT8 models are currently supported only by the CPU plugin. For the full list of supported configurations, see the [Supported Devices](./docs/IE_DG/supported_plugins/Supported_Devices.md) topic.
-## Hardware requirements
-Hardware requirements depend on a model. Typically for public models RAM memory size has to be not less then 16Gb, drive has to have not less then 30 GB free space independently on operation system. Temporary directory is used to cache layers output during calibration.
+You can run the Calibration Tool in two modes:
+
+* The **standard mode** performs quantization with the minimal accuracy drop within specified threshold compared to accuracy of the original model. This mode utilizes the [Accuracy Checker tool](./tools/accuracy_checker/README.md) to measure accuracy during the calibration process. Use this mode to obtain an INT8 IR that can be directly used in your application
+
+* The **simplified mode** produces the IR that contains plain statistics for each layer which is collected without any accuracy check, meaning that the accuracy of the new IR with statistics might be dramatically low. Therefore, all layers are considered to be executed in INT8. Use this mode to understand a potential performance gain of model conversion to INT8 precision and make a conclusion about running the standard mode routine.
+
+The Calibration Tool is a Python\* command-line tool, which imports Python types from the `openvino.tools.calibration` package.
+
+## System Requirements
+Hardware requirements depend on a model. Typically for public models RAM memory size has to be not less then 16GB, drive has to have not less then 30 GB free space independently on operation system. Temporary directory is used to cache layers output during calibration.
## Usage
-The Calibration Tool is configured in the same way as the Accuracy Checker. You can also use additional command-line arguments to define calibration-specific parameters.
+You can run the Calibration Tool in either standard or simplified mode with an appropriate set of configuration parameters.
+
+### Standard Mode
+In the standard mode, the Calibration Tool is configured in the same way as the Accuracy Checker.
+
+> **NOTE**: For consistency reasons, a part of arguments have the same name and meaning as in the Accuracy Checker and can be reused for running the Accuracy Checker.
+
+For configuring the tool, you can use the following command-line arguments:
+
+**Command-Line arguments common for the Calibration Tool and Accuracy Checker**
-### Command-Line Arguments for the Accuracy Checker Tool reused in Calibration Tool
| Argument | Type | Description |
| -------------------------------------------- | ------ | ------------------------------------------------------- |
-| -c, --config | string | Optional. Path to the YML file with local configuration |
-| -d, --definitions | string | Optional. Path to the YML file with definitions |
-| -m, --models | string | Optional. Prefix path to the models and weights. In the simplified mode, it is the path to IR .xml file |
-| -s, --source | string | Optional. Prefix path to the data source. In the simplified mode, it is the path to a folder with images |
-| -a, --annotations | string | Optional. Prefix path to the converted annotations and datasets meta data |
-| -e, --extensions | string | Optional. Prefix path to extensions folder. In simplified mode is a path to extensions library |
-| --cpu_extensions_mode, --cpu-extensions-mode | string | Optional. specified preferable set of processor instruction for automatic searching the CPU extension lib: `avx2` or `sse4` |
-| -C, --converted_models, --converted-models | string | Optional. Directory to store Model Optimizer converted models. Used for DLSDK launcher only |
-| -M, --model_optimizer, --model-optimizer | string | Optional. Path to model optimizer Caffe* directory |
-| --tf_custom_op_config_dir, --tf-custom-op-config-dir | string | Optional. Path to directory with TensorFlow* custom operation configuration files for model optimizer |
-| --tf_obj_detection_api_pipeline_config_path, --tf-obj-detection-api-pipeline-config-path | string | Optional. Path to directory with TensorFlow object detection API pipeline configuration files for the Model Optimizer |
-| --progress | string | Optional. Progress reporter: `bar`, `print` or `None` |
-| -td, --target_devices, --target-devices | string | Optional. Space-separated list of devices for infer |
-| -tt, --target_tags, --target-tags | string | Optional. Space-separated list of launcher tags for infer |
-
-### Specific Command Line Arguments for Calibration Tool
+| `-c`, `--config` | string | Required. Path to the YML file with local configuration. |
+| `-d`, `--definitions` | string | Optional. Path to the YML file with definitions. |
+| `-m`, `--models` | string | Optional. Prefix path to the models and weights. |
+| `-s`, `--source` | string | Optional. Prefix path to the data source. |
+| `-a`, `--annotations` | string | Optional. Prefix path to the converted annotations and datasets meta data. |
+| `-e`, `--extensions` | string | Optional. Prefix path to extensions folder. |
+| `--cpu_extensions_mode`, `--cpu-extensions-mode` | string | Optional. Preferable set of processor instruction for automatic searching the CPU extension lib: `avx2` or `sse4`. |
+| `-C`, `--converted_models`, `--converted-models` | string | Optional. Directory to store Model Optimizer converted models.|
+| `-M`, `--model_optimizer`, `--model-optimizer` | string | Optional. Path to model optimizer Caffe* directory. |
+| `--tf_custom_op_config_dir`, `--tf-custom-op-config-dir` | string | Optional. Path to directory with TensorFlow* custom operation configuration files for model optimizer. |
+| `--tf_obj_detection_api_pipeline_config_path`, `--tf-obj-detection-api-pipeline-config-path` | string | Optional. Path to directory with TensorFlow object detection API pipeline configuration files for the Model Optimizer. |
+| `--progress` | string | Optional. Progress reporter: `bar`, `print` or `None` |
+| `-td`, `--target_devices`, `--target-devices` | string | Optional. Space-separated list of devices for infer |
+| `-tt`, `--target_tags`, `--target-tags` | string | Optional. Space-separated list of launcher tags for infer |
+
+**Command Line Arguments specific for Calibration Tool**
+
| Argument | Type | Description |
| --------------------------------- | ------ | --------------------------------------------------------- |
-| -p, --precision | string | Optional. Precision to calibrate. Default value is INT8. In the simplified mode, determines output IR precision |
-| --ignore_layer_types, --ignore-layer-types | string | Optional. Layer types list which will be skipped during quantization |
-| --ignore_layer_types_path, --ignore-layer-types-path | string | Optional. Ignore layer types file path |
-| --ignore_layer_names, --ignore-layer-names | string | Optional. Layer names list which will be skipped during quantization |
-| --ignore_layer_names_path, --ignore-layer-names-path | string | Optional. Ignore layer names file path |
-| --batch_size, --batch-size | integer| Optional. Batch size value. If not specified, the batch size value is determined from IR |
-| -th, --threshold | float | Optional. Accuracy drop of quantized model should not exceed this threshold. Should be pointer in percents without percent sign. (1% is default) |
-| -ic, --benchmark_iterations_count, --benchmark-iterations-count | integer | Optional. Benchmark iterations count (1 is default). |
-| -mn, --metric_name, --metric-name | string | Optional. Metric name used during calibration |
-| -mt, --metric_type, --metric-type | string | Optional. Metric type used during calibration |
-| -o, --output_dir, --output-dir | string | Optional. Directory to store converted models. Original model directory is used if not defined |
-
-### Simplified mode
+| `-p`, `--precision` | string | Optional. Precision to calibrate. Default value is INT8. In the simplified mode, determines output IR precision. |
+| `--ignore_layer_types`, `--ignore-layer-types` | string | Optional. Layer types list which will be skipped during quantization. |
+| `--ignore_layer_types_path`, `--ignore-layer-types-path` | string | Optional. Ignore layer types file path. |
+| `--ignore_layer_names`, `--ignore-layer-names` | string | Optional. Layer names list which will be skipped during quantization. |
+| `--ignore_layer_names_path`, `--ignore-layer-names-path` | string | Optional. Ignore layer names file path. |
+| `--batch_size`, `--batch-size` | integer| Optional. Batch size value. If not specified, the batch size value is determined from IR. |
+| `-th`, `--threshold` | float | Optional. Accuracy drop of quantized model should not exceed this threshold. Should be pointer in percents without percent sign. (1% is default). |
+| `-ic`, `--benchmark_iterations_count`, `--benchmark-iterations-count` | integer | Optional. Benchmark iterations count (1 is default). |
+| `-mn`, `--metric_name`, `--metric-name` | string | Optional. Metric name used during calibration. |
+| `-mt`, `--metric_type`, `--metric-type` | string | Optional. Metric type used during calibration. |
+| `-o`, `--output_dir`, `--output-dir` | string | Optional. Directory to store converted models. Original model directory is used if not defined. |
+
+### Simplified Mode
+
+The tool in this mode does not use the Accuracy Checker, configuration and annotation files, but you are required to specify paths to an IR .xml file and a dataset folder. Optionally, you can specify a prefix path to an extensions folder and the number of images from the dataset folder:
+
| Argument | Type | Description |
| --------------------------------- | ------ | --------------------------------------------------------- |
-| -sm, --simplified_mode, --simplified-mode | | Optional. If specified, the Calibration Tool collects statistics without searching for optimal data thresholds. |
-| -ss, --subset | integer | Optional. This option is used only with --simplified_mode. Specifies a number of images from a folder that is set using `-s` option. |
+| `-sm`, `--simplified_mode`, `--simplified-mode` | | Required. If specified, the Calibration Tool runs in the simplified mode to collects statistics without searching for optimal data thresholds. |
+| `-m` | string | Required. Path to the IR .xml file. |
+| `-s`, `--source` | string | Optional. Path to a folder with images. |
+| `-ss`, `--subset` | integer | Optional. This option is used only with `--simplified_mode`. Specifies a number of images from a folder that is set using `-s` option. |
+| `-e`, `--extensions` | string | Optional. Prefix path to extensions folder. |
+| `-td`, `--target_devices`, `--target-devices` | string | Optional. Space-separated list of devices for infer. |
+| `-p`, `--precision` | string | Optional. Precision to calibrate. Default value is INT8. In the simplified mode, determines output IR precision. |
+| `-o`, `--output_dir`, `--output-dir` | string | Optional. Directory to store converted models. Original model directory is used if not defined. |
-## Model Calibration Flow
+## Typical Workflow Samples (Standard Mode)
### Introduction
-The calibration tool read original FP32 model, calibration dataset and create low precision model. Low precision model has two differences from original model:
+The calibration tool reads original FP16 or FP32 models, calibration dataset and creates a low precision model. The low precision model has two differences from the original model:
1. Per channel statistics are defined. Statistics have minimum and maximum values for each layer and each channel. Model statistics are stored in Inference Engine intermediate representation file (IR) in XML format.
2. `quantization_level` layer attribute is defined. The attribute defines precision which is used during inference.
### Prerequisites
-* Model: Tensorflow\* Inception v1. You can download the model from here: https://github.com/tensorflow/models/tree/master/research/slim
+* Model: TensorFlow* Inception v1. You can download the model from here: https://github.com/tensorflow/models/tree/master/research/slim
* Dataset: ImageNet. You can download ImageNet from here: http://www.image-net.org/download.php
* YML configuration files: you can find YML configuration files and YML definition file which are used below in `configs` directory:
- `definitions.yml` - definition file
- - `inception_v1.yml` - configuration file for Tensorflow\* Inception v1 model
- - `ncf_config.yml` - configuration file for NCF model in OpenVINO\* Inference Engine Intermediate Representation format
- - `ssd_mobilenet_v1_coco.yml` - configuration file for Tensorflow\* SSD Mobilenet v1 model
- - `unet2d.yml` - configuration file for Unet2D mode in in OpenVINO\* Inference Engine Intermediate Representation format
+ - `inception_v1.yml` - configuration file for TensorFlow* Inception v1 model
+ - `ncf_config.yml` - configuration file for NCF model in OpenVINO Inference Engine Intermediate Representation format
+ - `ssd_mobilenet_v1_coco.yml` - configuration file for TensorFlow* SSD Mobilenet v1 model
+ - `unet2d.yml` - configuration file for Unet2D mode in in OpenVINO Inference Engine Intermediate Representation format
-If you have custom topology with not supported accuracy metric or not suported custom dataset then you should add some components implementation in `openvino.tools.accuracy_checker` Python\* package yourself. Refer to `openvino.tools.accuracy_checker` documentation how to implement metric and dataset support.
+If your custom topology does not support accuracy metric or a custom dataset, add some components implementation in `openvino.tools.accuracy_checker` Python\* package yourself. For more information about metric implementation and dataset support, go to the [Accuracy Checker documentation](./tools/accuracy_checker/README.md).
There are steps to calibrate and evaluate result model:
-- Step #1. Convert data annotation files
-- Optional step for low precision model performance estimation.
-- Step #2. Calibration
-- Step #3. Result model evaluation
+1. Convert data annotation files.
+2. (Optional) Estimate low precision model performance.
+3. Calibrate the model.
+4. Evaluate the resulting model.
Additional optional step before calibration is available to rough estimate possible INT8 performance.
-### Step #1. Convert Data Annotation Files
+### Convert Data Annotation Files
Calibration dataset is subset of training dataset. Use Convert Annotation Tool to convert ImageNet\* dataset to Calibration Tool readable data annotation files. Data annotation files describe subset of images which are used during calibration. Command line:
```sh
python convert_annotation.py imagenet --annotation_file /datasets/ImageNet/val.txt --labels_file /datasets/ImageNet/synset_words.txt -ss 2000 -o ~/annotations -a imagenet.pickle -m imagenet.json
```
-> **NOTE:** For simplicity all command line tools in below steps use the same command line arguments. In practice [Collect Statistics Tool](./inference-engine/tools/collect_statistics_tool/README.md) uses calibration dataset, but [Accuracy Checker Tool](./inference-engine/tools/accuracy_checker_tool/README.md) has to use whole validation dataset.
+> **NOTE:** For simplicity, all command line tools in the steps below use the same command line arguments. In practice [Collect Statistics Tool](./inference-engine/tools/collect_statistics_tool/README.md) uses calibration dataset, but [Accuracy Checker Tool](./tools/accuracy_checker/README.md) has to use the whole validation dataset.
| Argument | Type | Description |
| -M | string | Path to model optimizer directory |
| --models | string | Prefix path to the models and weights |
| --source | string | Prefix path to the data source |
-| --annotations | string | Pefix path to the converted annotations and datasets meta data |
-| --converted_models | string | Directory to store Model Optimizer converted models. Used for DLSDK launcher only |
+| --annotations | string | Prefix path to the converted annotations and datasets meta data |
+| --converted_models | string | Directory to store Model Optimizer converted models |
-### Optional Step for Low Precision Model Performance Estimation
+### (Optional) Estimate Low-Precision Model Performance
Before calibration, you can roughly estimate low precision performance with [Collect Statistics Tool](./inference-engine/tools/collect_statistics_tool/README.md).
python collect_statistics.py --config ~/inception_v1.yml -d ~/defenitions.yml -M /home/user/intel/openvino/deployment_tools/model_optimizer --models ~/models --source /media/user/calibration/datasets --annotations ~/annotations --converted_models ~/models
```
-Result model has statistics which allow you to infer this model in INT8 precision. To measure performance you can use [Benchmark Tool](./inference-engine/tools/benchmark_tool/README.md).
+Result model has statistics which allow you to infer this model in INT8 precision. To measure performance, you can use the [Benchmark App](./inference-engine/ie_bridges/python/sample/benchmark_app/README.md).
-### Step #2. Calibration
-During calibration process, the model is ajusted for efficient quantization and minimization of accuracy drop on calibration dataset. Calibration tool produces calibrated model which will be executed in low precision 8 bit quantzed mode after loading into CPU plugin.
+### Calibrate the Model
+During calibration process, the model is adjusted for efficient quantization and minimization of accuracy drop on calibration dataset. Calibration tool produces calibrated model which will be executed in low precision 8-bit quantized mode after loading into CPU plugin.
[Calibration Tool](./inference-engine/tools/calibration_tool/README.md) has flexible and extensible mechanism of enabling new data set and metrics. Each network has its own dedicated network metric and dataset where network was trained. Dataset description and network metrics can be reused for different network.
-To plug new dataset you need to develop YML file. To develop new metric you need to develop Python\* module implementing metric and describe in YML. Please, refer to [Accuracy Checker Tool](./inference-engine/tools/accuracy_checker_tool/README.md) for details.
+To plug new dataset you need to develop YML file. To develop new metric you need to develop Python\* module implementing metric and describe in YML. Please, refer to [Accuracy Checker Tool](./tools/accuracy_checker/README.md) for details.
Command line example:
python calibrate.py --config ~/inception_v1.yml --definition ~/defenitions.yml -M /home/user/intel/openvino/deployment_tools/model_optimizer --tf_custom_op_config_dir ~/tf_custom_op_configs --models ~/models --source /media/user/calibration/datasets --annotations ~/annotations
```
-### Step #3. Result model evaluation
-After calibration of the model it worse to evaluate network accuracy on whole validation set using [Accuracy Checker Tool](./inference-engine/tools/accuracy_checker_tool/README.md).
+### Evaluate the Resulting Model
+After calibration of the model it worse to evaluate network accuracy on whole validation set using [Accuracy Checker Tool](./tools/accuracy_checker/README.md).
-#### Step #3.1 Check accuracy
+#### Check accuracy
Command line:
```sh
python accuracy_check.py --config ~/inception_v1.yml -d ~/defenitions.yml -M /home/user/intel/openvino/deployment_tools/model_optimizer --tf_custom_op_config_dir ~/tf_custom_op_configs --models ~/models --source /media/user/calibration/datasets --annotations ~/annotations -tf dlsdk -td CPU
```
-#### Step #3.2 Check performance
-Use `benchmark_app` command line tool to measure latency and throughput for synchronous and asynchronous modes. Note, please, `benchmark_app` command line tool uses converted OpenVINO\* Intermediate Representation model.
+#### Check performance
+Use the [Benchmark App](./inference-engine/samples/benchmark_app/README.md) command line tool to measure latency and throughput for synchronous and asynchronous modes. Note, the Benchmark App command line tool uses converted OpenVINO* Intermediate Representation model.
Command line for synchronous mode:
./benchmark_app -i <path_to_image>/inputImage.bmp -m <path_to_model>/inception_v1.xml -d CPU -api async
```
-#### Optional step to check performance
-You can use Python\* [Benchmark Tool](./inference-engine/tools/benchmark_tool/README.md) command line tool to quickly check performance with the same command line arguments and configuration YML files as for [Calibration Tool](./inference-engine/tools/calibration_tool/README.md).
-
-Command line:
-```sh
-python benchmark.py --config ~/inception_v1.yml -d ~/defenitions.yml -M /home/user/intel/openvino/deployment_tools/model_optimizer --tf_custom_op_config_dir ~/tf_custom_op_configs --models ~/models --source /media/user/calibration/datasets --annotations ~/annotations --converted_models ~/models
-```
-
-## Simplified Mode Flow
-
-The Calibration Tool in the simplified mode helps to quickly estimate performance of a model. It converts all possible layers into INT8 and collects statistics without achieving needed accuracy. The tool generates new IR, which is used in performance tests. Therefore, the tool in this mode does not use Accuracy Checker, configuration and annotation files, but you should specify paths to an IR .xml file and a dataset folder. Optionally, you can specify a path to an extensions library and the number of images from the dataset folder. In simplified mode path to an extensions is a path to extensions library.
+## Typical Workflow Samples (Simplified Mode)
To run the Calibration Tool in the simplified mode, use the following command:
```sh
-python3 calibrate.py -sm -m <path-to-ir.xml> -s <path-to-dataset> -ss img_num -e <path-to-extensions-library> -td target_device
+python3 calibrate.py -sm -m <path-to-ir.xml> -s <path-to-dataset> -ss <images-number> -e <path-to-extensions-folder> -td <target-device> -precision <output-ir-precision> --output-dir <output-directory-path>
```
-It accepts models with FP32, FP16 precisions and images as dataset.
-
+It accepts models with FP32, FP16 precisions and image files as the dataset.
\ No newline at end of file
projects / platform / upstream / dldt.git / blobdiff