# benchdnn
**benchdnn** is a standalone correctness and performance benchmark for
-[Intel(R) Math Kernel Library for Deep Neural Networks (Intel(R) MKL-DNN)](/intel/mkl-dnn) library.
+[Intel(R) Math Kernel Library for Deep Neural Networks (Intel(R) MKL-DNN)](/intel/mkl-dnn).
The purpose of the benchmark is extended and robust correctness verification of
-the primitives provided by MKL-DNN. So far **benchdnn** supports convolutions
-and inner products of different data types. It also implicitly tests reorders.
+the primitives provided by Intel MKL-DNN. Currently, **benchdnn** supports convolutions
+, inner products, reorder, batch normalization, deconvolution, recurrent neural network, and shuffle of different data types.
## License
## Usage (main driver)
-**benchdnn** itself is a driver for different implementation specific
-harnesses. So far it has harness for Intel MKL-DNN convolution, inner product,
-reorder, batch normalization, and harness for testing itself.
-The usage:
+**benchdnn** itself is a driver for different implementation-specific
+harnesses. So far it uses a harness for Intel MKL-DNN [convolution](/tests/benchdnn/README.md#usage-convolution-harness), [inner product](/tests/benchdnn/README.md#usage-ip-harness),
+[reorder](/tests/benchdnn/README.md#usage-reorder-harness), [batch normalization](/tests/benchdnn/README.md#usage-batch-normalization-harness), [deconvolution](/tests/benchdnn/README.md#usage-deconvolution-harness), [shuffle](/tests/benchdnn/README.md#usage-shuffle-harness), and [recurrent neural network](/tests/benchdnn/README.md#usage-rnn-harness) as well as a
+harness for testing [itself](/tests/benchdnn/README.md#usage-self-harness).
+
+Usage:
```
- $ ./benchdnn: [--HARNESS] [--mode=MODE] [-vN|--verbose=N] HARNESS-OPTS
+ $ ./benchdnn: [--HARNESS] [--mode=MODE] [--max-ms-per-prb=MAX-MS-PER-PRB] [-vN|--verbose=N] HARNESS-OPTS
```
where:
- - `HARNESS` is either `conv` [default], `ip`, `shuffle`, `reorder`, `bnorm`, `rnn` or `self`
+ - `HARNESS` is either `conv` [default], `ip`, `shuffle`, `reorder`, `bnorm`, `rnn`, or `self`
- `MODE` -- string that contains flags for benchmark mode. Use `C` or `c` for correctness (used by default), and `P` or `p` for performance
- - `N` -- verbose level (integer from 0 [default] to ...)
+ - `MAX-MS-PER-PRB` is passed to assign the maximum time spent per problem in milliseconds, by default `3e3`
+ - `-vN|--verbose=N` -- verbose level, default `0`
+
+ - `HARNESS-OPTS` are passed to the chosen harness
- - `HARNESS-OPTS` are passed to the chosen harness
+Returns `0` on success (all tests passed) or non-zero in case of any error.
-Returns `0` on success (all tests passed), and non-zero in case of any error
-happened.
+## Notations / Glossary / Abbreviations
+
+|Abbreviation | Description
+|:--- |:---
+| src | Source image (input image for forward convolution)
+| wei | Weights (aka filter)
+| bia | Bias
+| dst | Destination image (output image for forward convolution)
+| acc | Accumulation (typically in terms of data type)
+| ic, oc | Input/Output channels (aka feature maps)
+| ih, iw | Input height and width
+| oh, ow | Output height and width
+| kh, kw | Kernel (filter, weights) height and width
+| sh, sw | Convolution stride over height and width
+| ph, pw | Convolution top and left padding
+| mb | Minibatch (amount of images processed at once)
+| g | Groups (a way to reduce the amount of computations, see Alexnet topology)
+| FWD_{D,B} | forward w/o and w/ bias
+| BWD_{D,W,WB} | backward wrt data, weights, and weights and bias
+| DIRECT, WINO | convolution algorithm: direct or Winograd based
+| AUTO | convolution algorithm is chosen by MKL-DNN for best performance
## Usage (convolution harness)
-The usage:
```
[harness-knobs] [conv-desc] ...
```
where *harness-knobs* are:
- - `--cfg={f32, u8s8u8s32, ...}` configuration (see below), default `f32`
+ - `--cfg={f32, u8s8u8s32, ...}` configuration (see below [convolution configuration](/tests/benchdnn/README.md#convolution-configurations-also-known-as-precision-specification)), default `f32`
- `--dir={FWD_D (forward data), FWD_B (forward data + bias),FWD_I (forward data inference), BWD_D (backward data), BWD_W (backward weights), BWD_WB (backward weights + bias)}` direction, default `FWD_B`
- - `--alg={DIRECT, WINO}` convolution algorithm, default DIRECT
- - `--merge={NONE, RELU}` merged primitive, default NONE (nothing merged)
+ - `--alg={DIRECT, WINO, AUTO}` convolution algorithm, default DIRECT
- `--attr="attr_str"` convolution attributes (see in the section below), default `""` (no attributes set)
- `--mb=N` override minibatch that is specified in convolution description, default `0` (use mb specified in conv desc)
- `--match=regex` check only convolutions that match with regex, default is `".*"`. Notice: Windows may only interpret string arguments surrounded by double quotation marks.
- `--reset` reset all the parameters set before to default one
- `-vN|--verbose=N` verbose level, default `0`
- `--batch=file` use options from the given file (see in subdirectory)
+ - `--mode=` string that contains flags for benchmark mode. Use `C` or `c` for correctness (used by default), and `P` or `p` for performance
-and *conv-desc* is convolution description. The canonical form is:
+and *conv-desc* is the convolution description. The canonical form is:
```
gXmbXicXihXiwXocXohXowXkhXkwXshXswXphXpwXdhXdwXnS
```
-Here X is a number and S is string (n stands for name). Some of the parameters
-might be omitted if there is either default one (e.g. if g is not specified
-**benchdnn** uses 1) or if the can be computed automatically (e.g. output shape
-can be derived from the input one and kernel). Also if either width or height
-is not specified than it is assumed height == width. Special symbol `_` is
-ignored, hence maybe used as delimiter. See `str2desc()` in conv/conv_aux.cpp
-for more details and implicit rules :^)
+Here X is a number and S is a string (n stands for name). Some of the parameters
+may be omitted if a default exists (for example, if g is not specified
+**benchdnn** uses 1) or if it can be computed automatically (for example, the output shape
+can be derived from the input one and the kernel). Also, if either width or height
+is not specified, it is assumed that height == width. The special symbol `_` is
+ignored, so it may be used as a delimiter. See `str2desc()` in conv/conv_aux.cpp
+for more details and implicit rules.
-The attribute string *attr_str* is defined as (new lines for readability):
+The attribute string *attr_str* is defined as follows (line breaks are for readability):
```
[irmode={nearest,down};]
[oscale={none,common,per_oc}[:scale];]
Next, `oscale` stands for output_scales. The first parameter is the policy that
is defined below. The second optional parameter is a scale that specifies
-either the one common output scale (for `none` and `common` polices) or a
-starting point for `per_oc` policy, which uses many scales. The default scale
+either the one common output scale (for the `none` and `common` polices) or a
+starting point for the `per_oc` policy, which uses many scales. The default scale
is 1.0. Known policies are:
- `none` (default) means no output scales set (i.e. scale = 1.)
- `per_oc` corresponds to `mask=1<<1` (i.e. output channels) with different scale factors
Next, `post_ops` stands for post operation sequence. Currently supported post
-ops are:
+operations are:
- `relu` with no parameters (i.e. corresponding scale is 1., alg = eltwise_relu, alpha = beta = 0.)
- `sum` with optional parameter scale (default 1.)
-### convolution configurations (aka precision specification)
+### Convolution configurations (also known as precision specification)
`--cfg` option specifies what convolution would be used in terms of data type.
-Also it defines all the magic with data filling inside. For integer type
+Also it defines all the magic with data filling inside. For the integer type,
saturation is implicitly implied.
-Finally configuration defines threshold for computation errors (ideally we
-want keep it 0 and it seems to work for now).
+Finally configuration defines the threshold for computation errors (ideally we
+want to keep it at 0, and it seems to work for now).
The table below shows cases supported by Intel MKL-DNN and corresponding
configurations for **benchdnn**:
| s8 | s8 | u8 | s32 | s8s8u8s32 | same notes as for u8s8f32s32
-## Performance measurements
+### Performance measurements (convolution harness)
-**benchdnn** supports custom performance report. Template is passed via
+**benchdnn** supports a custom performance report. A template is passed via the
command line and consists of terminal and nonterminal symbols. Nonterminal
-symbols are printed as is. Description of terminal symbols is given below.
-There is also a notion of modifiers (marked as @) that change meaning of
-terminal symbols, e.g. sign '-' means minimum of (in terms of time). See
-table of modifiers below.
+symbols are printed as-is. A description of terminal symbols is given below.
+There is also a notion of modifiers (marked with @) that change the meaning of
+terminal symbols; for example, the sign '-' means minimum of (in terms of time).
+See the table of modifiers below.
-> **caution:** threads have to be pinned in order to get consistent frequency
+> **Caution:** Threads must be pinned in order to get consistent frequency.
-| abbreviation | description
+| Abbreviation | Description
|:------------ |:-----------
| %d | problem descriptor
| %D | expanded problem descriptor (conv parameters in csv format)
| %@c | time in clocks
| %@p | ops per second
-| modifier | description
+| Modifier | Description
|:-------- |:-----------
| | default
| - | min (time) -- default
The definition of expanded problem descriptor is:
`g,mb,ic,ih,iw,oc,oh,ow,kh,kw,sh,sw,ph,pw`.
-The default template can be found in conv/bench_conv.cpp that is defined as
+The default template can be found in conv/bench_conv.cpp and is defined as
`perf,%n,%d,%GO,%GF,%-t,%-Gp,%0t,%0Gp`. That will produce the following output
in CSV format:
```
average time spent in ms
average gigaops (since it corresponds to average time)
```
+Here is an example of the performance output:
+```
+ perf,"yolov2:conv1",mb16ic3ih610oc32oh608kh3n"yolov2:conv1",10.2205,0,43.9827,232.375,58.0146,176.171
+```
+full convolution descriptor is `mb16ic3ih610oc32oh608kh3n"yolov2:conv1"` in the above example.
-## Examples
+### Examples (convolution harness)
Run the set of f32 forward convolutions from inputs/conv_all file w/ bias and default minibatch:
```
--cfg=f32 --dir=FWD_B --batch=inputs/conv_all
```
-Run the same but with merged ReLU:
+Run the same but with post_ops ReLU:
```
$ ./benchdnn --conv \
- --cfg=f32 --dir=FWD_B --merge=RELU --batch=inputs/conv_all
+ --cfg=f32 --dir=FWD_B --attr="post_ops='relu'" --batch=inputs/conv_all
```
Run the same as previous but also measure performance:
```
- $ ./benchdnn --conv --mode=CORRnPERF \
- --cfg=f32 --dir=FWD_B --merge=RELU --batch=inputs/conv_all
+ $ ./benchdnn --conv --mode=CORRnPERF \
+ --cfg=f32 --dir=FWD_B --attr="post_ops='relu'" --batch=inputs/conv_all
```
-> **note**: instead of `CORRnPERF` one can use `CP`, `PC`, `cp`, or `pc`
+> **Note**: Instead of `CORRnPERF`, one can use `CP`, `PC`, `cp`, or `pc`
Run a set of f32 backward convolutions wrt weights with kh=3 and
verbose level set to 2:
--cfg=f32 ic3ih227iw227_oc96oh55ow55_kh11kw11_sh4sw4ph0pw0_n"alexnet:conv1"
```
-Run batch file for different algorithms (assuming the file only specifies
-convolutions and does not include harness options that would override ones
-passed in the command line). Also ignore mkldnn_unimplemented errors in case of
+Run batch file for different algorithms (assuming the file specifies only
+convolutions and does not include harness options that would override any
+passed on the command line). Also ignore mkldnn_unimplemented errors in case of
Winograd:
```
$ ./benchdnn --conv \
--alg=DIRECT --batch=convs.in \
--allow-unimpl=true \
- --alg=WINO --batch=convs.in
+ --alg=WINO --batch=convs.in \
+ --alg=AUTO --batch=convs.in
```
-Run a set of u8s8u8s32 forward convolutions w/o bias, skipping
+Run a set of u8s8u8s32 forward convolutions without bias, skipping
reference implementations and not triggering unimplemented as an error, with
one common output scale set to 0.5 with rounding mode set to down
(via attributes):
--attr="irmode=down;oscale=common:.5" --batch=inputs/conv_all
```
-Almost the same as above (with minor changes), but also add post operation
-sequence **(relu, then sum with scale .3, then relu)** using
-attributes/mkldnn_post_ops_t:
-```
- $ ./benchdnn --conv \
- --cfg=u8s8s32s32 --dir=FWD_B \
- --attr="oscale=common:.5;post_ops='relu;sum:.3;relu'" --batch=inputs/conv_all
-```
-
-
-## Notations / Glossary / Abbreviations
-
-|Abbreviation | Description
-|:--- |:---
-| src | Source image (input image for forward convolution)
-| wei | Weights (aka filter)
-| bia | Bias
-| dst | Destination image (output image for forward convolution)
-| acc | Accumulation (typically in terms of data type)
-| ic, oc | Input/Output channels (aka feature maps)
-| ih, iw | Input height and width
-| oh, ow | Output height and width
-| kh, kw | Kernel (filter, weights) height and width
-| sh, sw | Convolution stride over height and width
-| ph, pw | Convolution top and left padding
-| mb | Minibatch (amount of images processed at once)
-| g | Groups (a way to reduce the amount of computations, see Alexnet topology)
-| FWD_{D,B} | forward w/o and w/ bias
-| BWD_{D,W,WB} | backward wrt data, weights, and weights and bias
-| DIRECT, WINO | convolution algorithm: direct or Winograd based
-| NONE, RELU | merged primitives: nothing or ReLU
## Usage (batch normalization harness)
-The usage:
```
./benchdnn --bnorm [harness-knobs] bnorm-desc ...
```
- `--fmt={nchw, nChw16c, ...}` data layout, default `nchw`
- `--flags=[|G|S|R]` batch normalization flags, default `none` (G -- global stats, S -- use scale shift, R -- fuse with ReLU)
- `--attr="attr_str"` attributes (see in the convolution section above), default `""` (no attributes set)
- - `--match=regex` check only convolutions that match with regex, default is `".*"`. Notice: Windows may only interpret string arguments surrounded by double quotation marks.
+ - `--match=regex` check only bnorm that match with regex, default is `".*"`. Notice: Windows may only interpret string arguments surrounded by double quotation marks.
- `--skip-impl="str1[:str2]..."` skip implementation (see mkldnn_query_impl_info_str), default `""`
- `--perf-template=template-str` set template for performance report (very similar to the convolution one)
- `--reset` reset all the parameters set before to default one
and *bnorm-desc* is a batch normalization description. The canonical form is:
```
- mbXicXihXiwXepsYnS
+ mbXicXidXihXiwXepsYnS
```
-Here X is an integer number, Y is a real number, and S is string (n stands for
-name). Special symbol `_` is ignored, hence maybe used as delimiter. There are
+Here X is an integer number, Y is a real number, and S is a string (n stands for
+name). The special symbol `_` is ignored, so it may be used as delimiter. There are
some implicit rules:
- if mb is omitted set mb to 2
- if eps is omitted set eps to 1./16
+### Performance measurements (batch normalization harness)
+
+**benchdnn** supports a custom performance report. A template is passed via the
+command line and consists of terminal and nonterminal symbols. Nonterminal
+symbols are printed as-is. A description of terminal symbols is given below.
+There is also a notion of modifiers (marked with @) that change the meaning of
+terminal symbols; for example, the sign '-' means minimum of (in terms of time). See the
+table of modifiers below.
+
+> **Caution:** Threads must be pinned in order to get consistent frequency.
+
+| abbreviation | description
+|:------------ |:-----------
+| %d | problem descriptor
+| %D | expanded problem descriptor (parameters in csv format)
+| %n | problem name
+| %z | direction
+| %f | flags
+| %q | data type (precision)
+| %f | data format (layout)
+| %@t | time in ms
+
+The definition of expanded problem descriptor is: `mb,ic,id,ih,iw,eps`.
+
+The default template can be found in bnorm/bench_bnorm.cpp and is defined as
+`perf,%n,%z,%f,%q,%f,%D,%-t,%0t`. That will produce the following output
+in CSV format:
+```
+string: perf
+bnorm name
+direction
+batch normalization flags
+base data type
+batch normalization flags
+expanded bnorm problem descriptor
+minimum time spent in ms
+average time spent in ms
+```
+Here is an example of performance output:
+```
+perf,"resnet_50:bn_conv1",FWD_D,,f32,,50,64,1,112,112,0.0625,10.7729,77.1917
+```
+expanded bnorm problem descriptor is `50,64,1,112,112,0.0625` in the above example.
+
+### Examples (batch normalization harness)
+
+Run the set of bnorms from inputs/bnorm/bnorm_resnet_50 file with default minibatch:
+```
+ $ ./benchdnn --bnorm \
+ --batch=inputs/bnorm/bnorm_resnet_50
+```
+
+Run the same as previous but also measure performance:
+```
+ $ ./benchdnn --bnorm --mode=CORRnPERF \
+ --batch=inputs/bnorm/bnorm_resnet_50
+```
+
+
+## Usage (rnn harness)
+
+```
+ ./benchdnn --rnn [harness-knobs] [rnn-desc] ...
+```
+
+where *harness-knobs* are:
+
+ - `--prop={FWD_D (forward data), BWD_DW (backward data + weights)}` direction, default `FWD_D``
+ - `--alg={VANILLA_RNN, VANILLA_LSTM, VANILLA_GRU, LBR_GRU}` algorithm, default `VANILLA_RNN``
+ - `--direction={left2right, right2left, concat, sum}` direction, default `left2right``
+ - `--activation={RELU, LOGISTIC, TANH}` activation, default `RELU``
+ - `--reset` reset all the parameters set before to default one
+ - `--batch=file` use options from the given file (see in subdirectory)
+
+and *rnn-desc* is rnn description. The canonical form is:
+```
+ lXtXmbXsicXslcXdicXdlc
+```
+Here X is a number and S is a string. Some implicit rules:
+ - default values: l = 1, t = 1, mb = 2, S="wip"
+
+ - if slc/dlc/dic is undefined => slc/dlc/dic = sic
+
+See `str2desc()` in rnn/rnn_aux.cpp
+for more details and implicit rules :^)
+
+### Performance measurements (rnn harness)
+
+
+Runing rnn with performance measurememt mode will produce the following output
+in CSV format:
+```
+string: perf
+algorithm
+activation function
+direction
+expanded rnn problem descriptor
+name
+time spent in ms
+minimum time spent in ms
+maximum time spent in ms
+average time spent in ms
+```
+Here is an example of performance output:
+```
+perf,VANILLA_RNN,RELU,left2right,l1t1mb128sic512slc512dic512dlc512n""GNMT_enc-training"",time(ms):min=68.0007,max=176.006,avg=91.2686
+```
+expanded rnn problem descriptor is `l1t1mb128sic512slc512dic512dlc512n` in the above example.
+
+### Examples (rnn harness)
+
+Run the set of rnn training from inputs/rnn/rnn_training file with default minibatch:
+```
+ $ ./benchdnn --rnn \
+ --batch=inputs/rnn/rnn_training
+```
+
+Run the same as previous but also measure performance:
+```
+ $ ./benchdnn --rnn --mode=CORRnPERF \
+ --batch=inputs/rnn/rnn_training
+```
+
+
+## Usage (deconvolution harness)
+
+```
+ ./benchdnn --deconv [harness-knobs] [deconv-desc] ...
+```
+
+where *harness-knobs* are:
+
+ - `--cfg={f32, u8s8u8s32, ...}` configuration (ref conv session above [convolution configuration](/tests/benchdnn/README.md#convolution-configurations-also-known-as-precision-specification)), default `f32`
+ - `--match=regex` check only deconvolutions that match with regex, default is `".*"`. Notice: Windows may only interpret string arguments surrounded by double quotation marks.
+ - `--mb=N` override minibatch that is specified in deconvolution description, default `0` (use mb specified in deconv desc)
+ - `--dir={FWD_D (forward data), FWD_B (forward data + bias),FWD_I (forward data inference), BWD_D (backward data), BWD_W (backward weights), BWD_WB (backward weights + bias)}` direction, default `FWD_B`
+ - `--alg={DIRECT, WINO, AUTO}` deconvolution algorithm, default DIRECT
+ - `--attr="attr_str"` deconvolution attributes (see in the convolution section above), default `""` (no attributes set)
+ - `--skip-impl="str1[:str2]..."` skip implementation (see mkldnn_query_impl_info_str), default `""`
+ - `--allow-unimpl=true|false` do not treat unimplemented configuration as an error, default `false`
+ - `--perf-template=template-str` set template for performance report (see section *Performance measurements*)
+ - `--mode=` string that contains flags for benchmark mode. Use `C` or `c` for correctness (used by default), and `P` or `p` for performance
+ - `--reset` reset all the parameters set before to default one
+ - `-vN|--verbose=N` verbose level, default `0`
+ - `--batch=file` use options from the given file (see in subdirectory)
+
+and *deconv-desc* is deconvolution description. The canonical form is:
+```
+ gXmbXicXihXiwXocXohXowXkhXkwXshXswXphXpwXdhXdwXnS
+```
+Here X is a number and S is string (n stands for name). Some of the parameters
+might be omitted if a default exists (e.g. if g is not specified
+**benchdnn** uses 1) or if the can be computed automatically (e.g. output shape
+can be derived from the input one and kernel). Also if either width or height
+is not specified than it is assumed height == width. Special symbol `_` is
+ignored, hence maybe used as delimiter. See `str2desc()` in conv/conv_aux.cpp
+for more details and implicit rules :^)
+
+
+### Performance measurements (deconvolution harness)
+
+**benchdnn** supports a custom performance report. please refer above Performance measurements convolution harness session for detail, [convolution harness](/tests/benchdnn/README.md#performance-measurements-convolution-harness).
+
+The default template can be found in conv/bench_deconv.cpp and is defined as
+`perf,%n,%d,%GO,%GF,%-t,%-Gp,%0t,%0Gp`. That will produce the following output
+in CSV format:
+```
+string: perf
+deconvolution name
+full deconv-desc
+number of giga ops calculated
+effective cpu frequency in GHz (amb clocks[min] / time[min])
+minimum time spent in ms
+best gigaops (since it corresponds to mimimum time)
+average time spent in ms
+average gigaops (since it corresponds to average time)
+```
+Here is an example of performance output:
+```
+ perf,"alexnet:deconv1",mb256ic96ih55oc3oh227kh11sh4n"alexnet:deconv1",2.9733,0,249.474,11.9183,307.702,9.66291
+```
+full deconvolution descriptor is `mb256ic96ih55oc3oh227kh11sh4n"alexnet:deconv1"` in the above example.
+
+### Examples (deconvolution harness)
+
+Run the set of f32 forward deconvolutions from inputs/deconv_all file w/ bias and default minibatch:
+```
+ $ ./benchdnn --deconv \
+ --cfg=f32 --dir=FWD_B --batch=inputs/deconv_all
+```
+
+Run the same as previous but also measure performance:
+```
+ $ ./benchdnn --deconv --mode=CORRnPERF \
+ --cfg=f32 --dir=FWD_B --batch=inputs/deconv_all
+```
+
+## Usage (ip harness)
+
+```
+ ./benchdnn --ip [harness-knobs] [ip-desc] ...
+```
+
+where *harness-knobs* are:
+
+ - `--cfg={f32, u8s8u8s32, ...}` configuration (ref conv session above [convolution configuration](/tests/benchdnn/README.md#convolution-configurations-also-known-as-precision-specification)), default `f32``
+ - `--mb=N` override minibatch that is specified in ip description, default `0` (use mb specified in ip desc)
+ - `--dir={FWD_D (forward data), FWD_B (forward data + bias),FWD_I (forward data inference), BWD_D (backward data), BWD_W (backward weights), BWD_WB (backward weights + bias)}` direction, default `FWD_B`
+ - `--attr="attr_str"` ip attributes (see in the convolution section above), default `""` (no attributes set)
+ - `--allow-unimpl=true|false` do not treat unimplemented configuration as an error, default `false`
+ - `--perf-template=template-str` set template for performance report (see section *Performance measurements*)
+ - `--mode=` string that contains flags for benchmark mode. Use `C` or `c` for correctness (used by default), and `P` or `p` for performance
+ - `--reset` reset all the parameters set before to default one
+ - `-vN|--verbose=N` verbose level, default `0`
+ - `--batch=file` use options from the given file (see in subdirectory)
+
+and *ip-desc* is ip description. The canonical form is:
+```
+ mbXicXidXihXiwXSocXnS
+```
+Here X is a number and S is a string (n stands for name).
+The special symbol `_` is ignored, so it may be used as a delimiter.
+Some implicit rules:
+ - default values: mb = 2, id = 1, S="wip"
+
+ - if H is undefined => H = W
+
+ - if W is undefined => W = H
+
+See `str2desc()` in ip/ip_aux.cpp
+for more details and implicit rules :^)
+
+### Performance measurements (ip harness)
+
+**benchdnn** supports a custom performance report. A template is passed via the
+command line and consists of terminal and nonterminal symbols. Nonterminal
+symbols are printed as-is. A description of terminal symbols is given below.
+There is also a notion of modifiers (marked with @) that change the meaning of
+terminal symbols; for example, the sign '-' means minimum of (in terms of time). See the
+table of modifiers below.
+
+> **Caution:** Threads must be pinned in order to get consistent frequency.
+
+| abbreviation | description
+|:------------ |:-----------
+| %d | problem descriptor
+| %D | expanded problem descriptor (parameters in csv format)
+| %n | problem name
+| %z | direction
+| %f | flags
+| %q | data type (precision)
+| %f | data format (layout)
+| %@t | time in ms
+
+The definition of expanded problem descriptor is: `mb,oc,ic,id,ih,iw`.
+
+The default template can be found in bnorm/bench_ip.cpp and is defined as
+`perf,%D,%n,%z,%q,%-t,%-Gp,%0t,%0Gp`. That will produce the following output
+in CSV format:
+```
+string: perf
+expanded ip problem descriptor
+name
+direction
+data type
+minimum time spent in ms
+best gigaops (since it corresponds to mimimum time)
+average time spent in ms
+average gigaops (since it corresponds to average time)
+```
+
+Here is an example of performance output:
+```
+perf,112,1000,2048,1,1,1,"resnet:ip1",FWD_B,f32,3.99976,114.695,19.0323,24.1039
+```
+expanded ip problem descriptor is `112,1000,2048,1,1,1` in the above example.
+
+### Examples (ip harness)
+
+Run the set of ip from inputs/ip/ip_all file with default minibatch:
+```
+ $ ./benchdnn --ip \
+ --batch=inputs/ip/ip_all
+```
+
+Run the same as previous but also measure performance:
+```
+ $ ./benchdnn --ip --mode=CORRnPERF \
+ --batch=inputs/ip/ip_all
+```
+
+## Usage (shuffle harness)
+
+```
+ ./benchdnn --shuffle [harness-knobs] [dim]...
+```
+
+where *harness-knobs* are:
+
+ - `--match==regex` check only shuffle that match with regex, default is `".*"`. Notice: Windows may only interpret string arguments surrounded by double quotation marks.
+ - `--dir={FWD_D (forward data), FWD_B (forward data + bias),FWD_I (forward data inference), BWD_D (backward data), BWD_W (backward weights), BWD_WB (backward weights + bias)}` direction, default `FWD_B`
+ - `--dt={f32, s32, ...}` base data type, default `f32`
+ - `--fmt={nchw, nChw16c, ...}` data layout, default `nchw`
+ - `--axis=` default `1`
+ - `--group=` default `1`
+ - `--mode=` string that contains flags for benchmark mode. Use `C` or `c` for correctness (used by default), and `P` or `p` for performance
+ - `-vN|--verbose=N` verbose level, default `0`
+ - `--batch=file` use options from the given file (see in subdirectory)
+
+and *dim* is ip description. The canonical form is:
+```
+ dxdxdxdxd
+```
+Here d is a number.
+
+See `str2dims()` in shuffle/shuffle_aux.cpp for more details.
+
+### Performance measurements (shuffle harness)
+
+**benchdnn** supports a custom performance report. A template is passed via the
+command line and consists of terminal and nonterminal symbols. Nonterminal
+symbols are printed as-is. A description of terminal symbols is given below.
+There is also a notion of modifiers (marked with @) that change the meaning of
+terminal symbols; for example, the sign '-' means minimum of (in terms of time). See the
+table of modifiers below.
+
+> **Caution:** Threads must be pinned in order to get consistent frequency.
+
+| Abbreviation | Description
+|:------------ |:-----------
+| %d | problem descriptor
+| %D | expanded problem descriptor (parameters in csv format)
+| %z | direction
+| %q | data type (precision)
+| %f | data format (layout)
+| %a | axis
+| %g | group size
+| %@t | time in ms
+
+The definition of expanded problem descriptor is: `dxdxdxdxd`.
+
+The default template can be found in shuffle/bench_shuffle.cpp and is defined as
+`perf,%z,%q,%f,%D,%a,%g,%-t,%0t`. That will produce the following output
+in CSV format:
+```
+string: perf
+direction
+data type
+data format
+expanded shuffle problem descriptor
+axis
+group size
+minimum time spent in ms
+average time spent in ms
+```
+Here is an example of performance output.
+```
+perf,FWD_D,u8,nCdhw16c,1x272x2x56x56,4,4,11.6177,16.509
+```
+expanded shuffle problem descriptor is `1x272x2x56x56` in the above example.
+
+### Examples (shuffle harness)
+
+Run the set of shuffle from inputs/shuffle/test_shuffle_axis file with default minibatch:
+```
+ $ ./benchdnn --shuffle \
+ --batch=inputs/shuffle/test_shuffle_axis
+```
+
+Run the same as previous but also measure performance:
+```
+ $ ./benchdnn --shuffle --mode=CORRnPERF \
+ --batch=inputs/shuffle/test_shuffle_axis
+```
+
+## Usage (reorder harness)
+
+```
+ ./benchdnn --reorder [harness-knobs] ...
+```
+
+where *harness-knobs* are:
+
+ - `--idt={f32, s32, ...}` base input data type, default `f32`
+ - `--odt={f32, s32, ...}` base output data type, default `f32`
+ - `--dt={f32, s32, ...}` base data type, default `f32`
+ - `--ifmt={nchw, nChw16c, ...}` input data layout, default `nchw`
+ - `--ofmt={nchw, nChw16c, ...}` output data layout, default `nchw`
+ - `--fmt={nchw, nChw16c, ...}` data layout, default `nchw`
+ - `--def-scales={,,}` input defined scales. separate number by ',' ex : 0.125, 0.25, 0.5, 1, 2, 4, 8
+ - `--attr="attr_str"` ip attributes (see in the section below), default `""` (no attributes set)
+ - `--both-dir-dt=true|false` , default `false`
+ - `--both-dir-fmt=true|false` , default `false`
+ - `--allow-unimpl=true|false` do not treat unimplemented configuration as an error, default `false`
+ - `--run` run reorder bench
+ - `--perf-template=template-str` set template for performance report (see section *Performance measurements*)
+ - `--reset` reset all the parameters set before to default one
+ - `--mode=` string that contains flags for benchmark mode. Use `C` or `c` for correctness (used by default), and `P` or `p` for performance
+ - `-vN|--verbose=N` verbose level, default `0`
+ - `--batch=file` use options from the given file (see in subdirectory)
+
+### Performance measurements (reorder harness)
+
+**benchdnn** supports a custom performance report. A template is passed via the
+command line and consists of terminal and nonterminal symbols. Nonterminal
+symbols are printed as-is. A description of terminal symbols is given below.
+There is also a notion of modifiers (marked with @) that change the meaning of
+terminal symbols; for example, the sign '-' means minimum of (in terms of time). See the
+table of modifiers below.
+
+> **Caution:** Threads must be pinned in order to get consistent frequency.
+
+| abbreviation | description
+|:------------ |:-----------
+| %d | problem descriptor
+| %D | expanded problem descriptor (reorder parameters in csv format)
+| %n | dimensionality of the problem
+| %@O | number of elements being reordered
+| %@t | time in ms
+| %@p | elements per second
+
+| modifier | description
+|:-------- |:-----------
+| | default
+| - | min (time) -- default
+| 0 | avg (time)
+| + | max (time)
+| |
+| K | Kilo (1e3)
+| M | Mega (1e6)
+| G | Giga (1e9)
+
+The definition of expanded problem descriptor is:
+`idt,odt,ifmt,ofmt,attrs,dims`.
+
+The default template can be found in reorder/bench_reorder.cpp and is defined as
+`perf,%n,%D,%O,%-t,%-Gp,%0t,%0Gp`. That will produce the following output
+in CSV format:
+```
+string: perf
+dimensionality of the problem
+expanded reorder problem descriptor
+number of elements being reordered
+minimum time spent in ms
+best gigaops (since it corresponds to mimimum time)
+average time spent in ms
+average gigaops (since it corresponds to average time)
+```
+Here is an example of performance output:
+```
+ perf,4,f32,f32,nchw,nchw,irmode=nearest;oscale=per_oc:0.125;post_ops='',2x64x3x3,1152,4.00244,0.000287824,24.0279,4.79442e-05
+```
+expanded reorder problem descriptor is `f32,f32,nchw,nchw,irmode=nearest;oscale=per_oc:0.125;post_ops='',2x64x3x3` in the above example.
+
+### Examples (reorder harness)
+
+Run the set of reorder from reorder/test_default file with default minibatch:
+```
+ $ ./benchdnn --reorder \
+ --batch=inputs/reorder/test_default
+```
+
+Run the same as previous but also measure performance:
+```
+ $ ./benchdnn --reorder --mode=CORRnPERF \
+ --batch=inputs/reorder/test_default
+```
+
+## Usage (self harness)
+
+```
+ ./benchdnn --self ...
+```
+
+Check enumlation type, attributes, flags, and descriptions.
+
+
## Installation
-**benchdnn** is automatically built with Intel MKL-DNN. For the convenience one
-may build **benchdnn** using cmake or make.
+**benchdnn** is automatically built with Intel MKL-DNN. For convenience, you can
+build **benchdnn** using cmake or make.
## Essence of convolution testing
-Intel MKL-DNN supports different data types, such as single precision floating
-point (`mkldnn_f32`), signed/unsigned integer of different length
-(`mkldnn_{s,u}{8,16,32}`). We need to cover all those cases by tests. It is
-essential to test real convolution sizes, since Intel MKL-DNN provides
-different optimizations depending on convolution parameters, so there is no
-one unified approach inside, which means it would not be enough to test only
-few convolutions (aka unit tests).
-
-But even for given convolution the correctness convolution test is not as
-simple as it might seem to be at first sight. One of the biggest problem we
-encountered is numerical instability. For every output point a lot of
-operations may happen. For instance on backward propagation with respect to
-filter each filter point requires `mb * oh * ow` operations (see *Notation*
-section below). That big amount of compute operations may lead to either
+Intel MKL-DNN supports different data types, such as single-precision floating
+point (`mkldnn_f32`) and signed/unsigned integer of different length
+(`mkldnn_{s,u}{8,16,32}`). We need to cover all those cases with tests. It is
+essential to test real convolution sizes, because Intel MKL-DNN provides
+different optimizations depending on convolution parameters. There is no
+single unified approach inside, so it would not be enough to test only a few
+convolutions (also known as unit tests).
+
+But even for a given convolution, the correctness convolution test is not as
+simple as it might seem at first sight. One of the biggest problems we
+encountered is numerical instability. For every output point, a lot of
+operations may occur. For instance, on backward propagation with respect to
+filter, each filter point requires `mb * oh * ow` operations (see the *Notation*
+section below). That large amount of compute operations may lead to either
integer overflow or accuracy loss if initial data was chosen inadequately.
-These two main things complicate testing. **benchdnn** tries to address these
-issues by using integers for initialization with uniform distribution in a
+These two main issues complicate testing. **benchdnn** tries to address these
+by using integers for initialization with uniform distribution in a
range `[cfg->f_min .. cfg->f_max]`, with the step `cfg->f_step`
(see `struct dt_conf_t` in conv/conv.hpp). `f_min` and `f_max` are chosen so
-that most of the result would belong `[cfg->min .. cfg->max]` range. Also
-for floating point all integers in both ranges have exact representation (i.e.
+that most of the results would belong in the `[cfg->min .. cfg->max]` range. Also
+for floating point all integers in both ranges have exact representation (that is,
the absolute numbers are less than `2^size_of_mantissa`). Uniform distribution
-leads to have result uniformly distributed and quite small `f_min/f_max` keep
+leads to results that are uniformly distributed and quite small. `f_min/f_max` keep
the result in a reasonable range. Yet another trick: not all the points are
initialized with non-zero values: see `fill_{src,wei,bia,dst}` in
conv/conv.cpp.
## Further plans
-Please see TODO.md in **benchdnn** root directory for development plans.
+Please see TODO.md in the **benchdnn** root directory for development plans.
## Issues and contributions
-We welcome community contributions to **benchdnn** as well as Intel MKL-DNN.
+We welcome community contributions to **benchdnn** as well as to Intel MKL-DNN.
If you have any ideas or issues please submit an issue or pull request. For
-clarity please include ''benchdnn: '' in the title.
+clarity, please include ''benchdnn: '' in the title.
## Inspiration
projects / platform / upstream / dldt.git / blobdiff