Skip to content

Commit be918b3

Browse files
timxu826timxu826
committed
first commit
0 parents  commit be918b3

10 files changed

+1507
-0
lines changed

‎README.md

+285
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,285 @@
1+
# Cross Check Tool {#openvino_inference_engine_tools_cross_check_tool_README}
2+
3+
Cross Check Tool is a console application that enables comparing accuracy and performance metrics for two successive
4+
model inferences that are performed on two different supported Intel® devices or with different precisions.
5+
The Cross Check Tool can compare the metrics per layer or all over the model.
6+
7+
## Running the Cross Check Tool
8+
9+
Cross Check Tool is distributed as a Python module and there is no need to build it. To run the Cross Check Tool,
10+
execute the `cross_check_tool.py` file with necessary parameters. Please note that the Inference Engine assumes that weights
11+
are in the same folder as the `.xml` file.
12+
13+
You can get the list of all available options using the `-h` option:
14+
15+
```sh
16+
$python3 cross_check_tool.py -h
17+
18+
Cross Check Tool is a console application that enables comparing accuracy and
19+
provides performance metrics
20+
21+
optional arguments:
22+
-h, --help show this help message and exit
23+
24+
Model specific arguments:
25+
--input INPUT, -i INPUT
26+
Path to an input image file or multi-input file to
27+
infer. Generates input(s) from normal distribution if
28+
empty
29+
--model MODEL, -m MODEL
30+
Path to an .xml file that represents the first IR of
31+
the trained model to infer.
32+
--reference_model REFERENCE_MODEL, -ref_m REFERENCE_MODEL
33+
Path to an .xml file that represents the second IR in
34+
different precision to compare the metrics.
35+
--layers LAYERS, -layers LAYERS
36+
Defines layers to check. Options: all, None - for
37+
output layers check, list of comma-separated layer
38+
names to check. Default value is None.
39+
--ref_layers REFERENCE_LAYERS, -reference_layers REFERENCE_LAYERS
40+
Defines layers to check in reference model. Options: all, None - for
41+
output layers check, list of comma-separated layer
42+
names to check. If not specified the same layers will
43+
be processed as in --layers parameter.
44+
--num_of_iterations NUM_OF_ITERATIONS, -ni NUM_OF_ITERATIONS
45+
Number of iterations to collect all over the net
46+
performance
47+
48+
Plugin specific arguments:
49+
--plugin_path PLUGIN_PATH, -pp PLUGIN_PATH
50+
Path to a plugin folder.
51+
--device DEVICE, -d DEVICE
52+
The first target device to infer the model specified
53+
with the -m or --model option. CPU, GPU or GNA are acceptable.
54+
--config CONFIG, -conf CONFIG
55+
Path to config file for -d or -device device plugin
56+
--reference_device REFERENCE_DEVICE, -ref_d REFERENCE_DEVICE
57+
The second target device to infer the model and
58+
compare the metrics. CPU, GPU or GNA are
59+
acceptable.
60+
--reference_config REFERENCE_CONFIG, -ref_conf REFERENCE_CONFIG
61+
Path to config file for -ref_d or -reference_device
62+
device plugin
63+
-l L Required for (CPU)-targeted custom layers.
64+
Comma separated paths to a shared libraries with the
65+
kernels implementation.
66+
67+
CCT mode arguments:
68+
--dump Enables blobs statistics dumping
69+
--load LOAD Path to a file to load blobs from
70+
71+
```
72+
73+
Cross Check Tool can also be installed via:
74+
```sh
75+
$python3 -m pip install <openvino_repo>/tools/cross_check_tool
76+
```
77+
In this case, to run the tool, call `cross_check_tool` on the command line with necessary parameters.
78+
79+
### Examples
80+
81+
1. To check per-layer accuracy and performance of inference in FP32 precision on the CPU against the GPU, run:
82+
```sh
83+
$python3 cross_check_tool.py -i <path_to_input_image_or_multi_input_file> \
84+
-m <path_to_FP32_xml> \
85+
-d GPU \
86+
-ref_d CPU \
87+
--layers all
88+
```
89+
90+
The output looks as follows:
91+
```sh
92+
[ INFO ] Cross check with one IR was enabled
93+
[ INFO ] GPU:FP32 vs CPU:FP32
94+
[ INFO ] The same IR on both devices: <path_to_IR>
95+
[ INFO ] Statistics will be dumped for X layers: <layer_1_name>, <layer_2_name>, ... , <layer_X_name>
96+
[ INFO ] Layer <layer_1_name> statistics
97+
Max absolute difference : 1.15204E-03
98+
Min absolute difference : 0.0
99+
Max relative difference : 1.15204E+17
100+
Min relative difference : 0.0
101+
Min reference value : -1.69513E+03
102+
Min absolute reference value : 2.71080E-06
103+
Max reference value : 1.17132E+03
104+
Max absolute reference value : 1.69513E+03
105+
Min actual value : -1.69513E+03
106+
Min absolute actual value : 8.66465E-05
107+
Max actual value : 1.17132E+03
108+
Max absolute actual value : 1.69513E+03
109+
Device: -d GPU -ref_d CPU
110+
Status: OPTIMIZED_OUT OPTIMIZED_OUT
111+
Layer type: Convolution Convolution
112+
Real time, microsec: 0 120
113+
Number of NAN: 0 0
114+
Number of INF: 0 0
115+
Number of ZERO: 0 0
116+
...
117+
<list_of_layer_statistics>
118+
...
119+
120+
[ INFO ] Overall max absolute difference = 0.00115203857421875
121+
[ INFO ] Overall min absolute difference = 0.0
122+
[ INFO ] Overall max relative difference = 1.1520386483093504e+17
123+
[ INFO ] Overall min relative difference = 0.0
124+
[ INFO ] Execution successful
125+
```
126+
127+
2. To check the overall accuracy and performance of inference on the CPU in FP32 precision against the
128+
Intel&reg; Movidius&trade; Myriad&trade; device in FP16 precision, run:
129+
```sh
130+
$python3 cross_check_tool.py -i <path_to_input_image_or_multi_input_file> \
131+
-m <path_to_FP16_xml> \
132+
-d MYRIAD \
133+
-ref_m <path_to_FP32_xml> \
134+
-ref_d CPU
135+
```
136+
137+
The output looks as follows:
138+
```sh
139+
[ INFO ] Cross check with two IRs was enabled
140+
[ INFO ] GPU:FP16 vs CPU:FP32
141+
[ INFO ] IR for MYRIAD : <path_to_FP16_xml>
142+
[ INFO ] IR for CPU : <path_to_FP32_xml>
143+
[ INFO ] Statistics will be dumped for 1 layer: <output_layer_name(s)>
144+
[ INFO ] Layer <output_layer_name> statistics
145+
Max absolute difference : 2.32944E-02
146+
Min absolute difference : 3.63002E-13
147+
Max relative difference : 6.41717E+10
148+
Min relative difference : 1.0
149+
Min reference value : 3.63002E-13
150+
Min absolute reference value : 3.63002E-13
151+
Max reference value : 7.38138E-01
152+
Max absolute reference value : 7.38138E-01
153+
Min actual value : 0.0
154+
Min absolute actual value : 0.0
155+
Max actual value : 7.14844E-01
156+
Max absolute actual value : 7.14844E-01
157+
Device: -d MYRIAD -ref_d CPU
158+
Status: OPTIMIZED_OUT OPTIMIZED_OUT
159+
Layer type: Reshape Reshape
160+
Real time, microsec: 0 0
161+
Number of NAN: 0 0
162+
Number of INF: 0 0
163+
Number of ZERO: 0 0
164+
----------------------------------------------------------------------
165+
Overall performance, microseconds: 2.79943E+05 6.24670E+04
166+
----------------------------------------------------------------------
167+
[ INFO ] Overall max absolute difference = 0.023294448852539062
168+
[ INFO ] Overall min absolute difference = 3.630019191052519e-13
169+
[ INFO ] Overall max relative difference = 64171696128.0
170+
[ INFO ] Overall min relative difference = 1.0
171+
[ INFO ] Execution successful
172+
```
173+
174+
3. To dump layer statistics from a specific list of layers, run:
175+
```sh
176+
$python3 cross_check_tool.py -i <path_to_input_image_or_multi_input_file> \
177+
-m <path_to_FP16_xml> \
178+
-d GNA \
179+
--dump \
180+
--layers <comma_separated_list_of_layers>
181+
```
182+
183+
The output looks as follows:
184+
```sh
185+
[ INFO ] Dump mode was enabled
186+
[ INFO ] <layer_1_name> layer processing
187+
...
188+
[ INFO ] <layer_X_name> layer processing
189+
[ INFO ] Dump file path: <path_where_dump_will_be_saved>
190+
[ INFO ] Execution successful
191+
```
192+
193+
If you do not provide the `-i` key, the Cross Check Tool generates an input from normal distributed noise and saves
194+
it in a multi-input file format with the filename `<path_to_xml>_input_layers_dump.txt` in the same folder as the Intermediate Representation (IR).
195+
196+
4. To check the overall accuracy and performance of inference on the CPU in FP32 precision against dumped results, run:
197+
```sh
198+
$python3 cross_check_tool.py -i <path_to_input_image_or_multi_input_file> \
199+
-m <path_to_FP32_xml> \
200+
-d CPU \
201+
--load <path_to_dump> \
202+
--layers all
203+
```
204+
205+
The output looks as follows:
206+
```sh
207+
[ INFO ] Load mode was enabled
208+
[ INFO ] IR for CPU : <path_to_FP32_xml>
209+
[ INFO ] Loading blob from /localdisk/models/FP16/icv_squeezenet_v1.0.xml_GPU_dump.npz
210+
[ INFO ] Statistics will be dumped for X layers: <layer_1_name>, <layer_2_name>, ... , <layer_X_name>
211+
[ INFO ] Layer <layer_1_name> statistics
212+
Max absolute difference : 0.0
213+
Min absolute difference : 0.0
214+
Max relative difference : 0.0
215+
Min relative difference : 0.0
216+
Min reference value : 0.0
217+
Min absolute reference value : 0.0
218+
Max reference value : 7.14844E-01
219+
Max absolute reference value : 7.14844E-01
220+
Min actual value : 0.0
221+
Min absolute actual value : 0.0
222+
Max actual value : 7.14844E-01
223+
Max absolute actual value : 7.14844E-01
224+
Device: -d CPU -load GPU
225+
Status: OPTIMIZED_OUT OPTIMIZED_OUT
226+
Layer type: Reshape Reshape
227+
Real time, microsec: 0 0
228+
Number of NAN: 0 0
229+
Number of INF: 0 0
230+
Number of ZERO: 609 699
231+
232+
...
233+
<list_of_layer_statistics>
234+
...
235+
236+
[ INFO ] Overall max absolute difference = 0.0
237+
[ INFO ] Overall min absolute difference = 0.0
238+
[ INFO ] Overall max relative difference = 0.0
239+
[ INFO ] Overall min relative difference = 0.0
240+
[ INFO ] Execution successful
241+
```
242+
243+
### Multi-input and dump file format
244+
245+
Multi-input and dump file is a numpy compressed `.npz` file with hierarchy:
246+
247+
```sh
248+
{
249+
‘layer_name’: {
250+
‘blob’: np.array([…])
251+
‘pc’: {
252+
‘device’: ‘device_name’,
253+
‘real_time’: int_real_time_in_microseconds_from_plugin,
254+
‘exec_type’: ‘exec_type_from_plugin’,
255+
‘layer_type’: ‘layer_type_from_plugin’,
256+
‘status’: ‘status_from_plugin’
257+
}
258+
},
259+
‘another_layer_name’: {
260+
‘blob’: np.array([…])
261+
‘pc’: {
262+
‘device’: ‘device_name’,
263+
‘real_time’: int_real_time_in_microseconds_from_plugin,
264+
‘exec_type’: ‘exec_type_from_plugin’,
265+
‘layer_type’: ‘layer_type_from_plugin’,
266+
‘status’: ‘status_from_plugin’
267+
}
268+
},
269+
...
270+
}
271+
```
272+
273+
### Configuration file
274+
275+
There is an option to pass configuration file to plugin by providing
276+
`--config` and/or `--reference_config` keys.
277+
278+
Configuration file is a text file with content of pairs of keys and values.
279+
280+
Structure of configuration file:
281+
282+
```sh
283+
KEY VALUE
284+
ANOTHER_KEY ANOTHER_VALUE,VALUE_1
285+
```

‎compare_realloc.sh

+29
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
#!/bin/bash
2+
3+
# Define the common variables
4+
set -exo
5+
Param_OV_GPU_Verbose="1"
6+
Param_OV_GPU_DumpLayersPath="/media/NewDisk/jia3xu/openvino/CVS-156289/issued/dump/layers/apply_27780a6/"
7+
Param_input_path="/media/NewDisk/jia3xu/openvino/CVS-156289/issued/data/inputs/data_2.pth"
8+
Param_model_path="/media/NewDisk/jia3xu/openvino/CVS-156289/issued/model/Pytorch_Minicpm_2B_api_2_True_batch_2_device_GPU_precision_FP16_ylyolf1/model.xml"
9+
Param_op_list_path="/media/NewDisk/jia3xu/openvino/CVS-156289/issued/dump/analysis_path/4227_mapping.pkl"
10+
11+
# Define the layers to dump
12+
#？
13+
# layers_to_dump=("fullyconnected:__module.model.layers.2.self_attn.k_proj/aten::linear/MatMul" "fullyconnected:__module.model.layers.2.self_attn.q_proj/aten::linear/MatMul" "fullyconnected:__module.model.layers.2.self_attn.v_proj/aten::linear/MatMul" "fullyconnected:__module.model.layers.2/aten::mul/Multiply_decompressed_to_f32" "fullyconnected:__module.model.layers.2.mlp.up_proj/aten::linear/MatMul" "fullyconnected:__module.model.layers.2.mlp.gate_proj/aten::linear/MatMul" "fullyconnected:__module.model.layers.2.mlp.gate_proj/aten::linear/MatMul" "fullyconnected:__module.model.layers.2/aten::mul/Multiply_1_decompressed_to_f32")
14+
layers_to_dump=("__module.model/aten::masked_fill/Select" "__module.model/aten::masked_fill/Select_1" "__module.model/aten::to/Convert_3")
15+
16+
# Loop through each layer and execute the command
17+
for layer in "${layers_to_dump[@]}"; do
18+
Param_OV_GPU_DumpLayers="$layer"
19+
log_file="log_$(echo $layer | tr -d ':/').txt" # Create a log filename from the layer name
20+
21+
# Execute the command and save the output to the log file
22+
OV_GPU_Verbose=$Param_OV_GPU_Verbose OV_GPU_DumpLayers=$Param_OV_GPU_DumpLayers \
23+
OV_GPU_DumpLayersPath=$Param_OV_GPU_DumpLayersPath \
24+
python ./cross_check_tool.py \
25+
-i $Param_input_path \
26+
-m $Param_model_path \
27+
-d GPU -ref_d CPU --layers file --op_list $Param_op_list_path \
28+
2>&1 | tee $log_file
29+
done

‎cross_check_tool.py

+11
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
#!/usr/bin/python3
2+
3+
# Copyright (C) 2018-2023 Intel Corporation
4+
# SPDX-License-Identifier: Apache-2.0
5+
6+
import sys
7+
from openvino.tools.cross_check_tool.cross_check_tool import main
8+
9+
10+
if __name__ == "__main__":
11+
sys.exit(main() or 0)

‎openvino/__init__.py

+3
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
# Copyright (C) 2018-2023 Intel Corporation
2+
# SPDX-License-Identifier: Apache-2.0
3+
__path__ = __import__('pkgutil').extend_path(__path__, __name__)

‎openvino/tools/__init__.py

+2
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
# Copyright (C) 2018-2023 Intel Corporation
2+
# SPDX-License-Identifier: Apache-2.0

‎openvino/tools/cross_check_tool/__init__.py

+2
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
# Copyright (C) 2018-2023 Intel Corporation
2+
# SPDX-License-Identifier: Apache-2.0

0 commit comments

Comments
 (0)