Skip to content

Commit 6de610c

Browse files
AlexKoff88AlexKoff88
committed
Fixed small issue. Results are the same.
1 parent b6220d5 commit 6de610c

File tree

1 file changed

+38
-44
lines changed

1 file changed

+38
-44
lines changed

‎notebooks/openvino/sentence_transformer_quantization.ipynb

+38-44
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,7 @@
4444
{
4545
"data": {
4646
"application/vnd.jupyter.widget-view+json": {
47-
"model_id": "fd8729d418f3453bb1d97a2b038ff072",
47+
"model_id": "53d4d1f1703a4e52812ea366c06f2d67",
4848
"version_major": 2,
4949
"version_minor": 0
5050
},
@@ -68,7 +68,7 @@
6868
{
6969
"data": {
7070
"application/vnd.jupyter.widget-view+json": {
71-
"model_id": "758502bfa6a142cc9078b8404a4b5d78",
71+
"model_id": "a3de9a9bbdd942069b85519c83267f83",
7272
"version_major": 2,
7373
"version_minor": 0
7474
},
@@ -92,7 +92,7 @@
9292
{
9393
"data": {
9494
"application/vnd.jupyter.widget-view+json": {
95-
"model_id": "e82786c713694f0da616dee6164aa242",
95+
"model_id": "ebc55f3ce3974aaa8861474699d5a15f",
9696
"version_major": 2,
9797
"version_minor": 0
9898
},
@@ -116,7 +116,7 @@
116116
{
117117
"data": {
118118
"application/vnd.jupyter.widget-view+json": {
119-
"model_id": "344f6318563c4bed8dbedbdee08c9b59",
119+
"model_id": "f206e4e8651f4f449f9dcb1fc11ef266",
120120
"version_major": 2,
121121
"version_minor": 0
122122
},
@@ -267,7 +267,7 @@
267267
"vanilla_emb = SentenceEmbeddingPipeline(model=model, tokenizer=tokenizer)\n",
268268
"\n",
269269
"q_model = OVModelForFeatureExtraction.from_pretrained(int8_ptq_model_path)\n",
270-
"q8_emb = SentenceEmbeddingPipeline(model=model, tokenizer=tokenizer)"
270+
"q8_emb = SentenceEmbeddingPipeline(model=q_model, tokenizer=tokenizer)"
271271
]
272272
},
273273
{
@@ -292,13 +292,13 @@
292292
"name": "stderr",
293293
"output_type": "stream",
294294
"text": [
295-
"Parameter 'function'=<function evaluate_stsb at 0x7f62282ab380> of the transform datasets.arrow_dataset.Dataset._map_single couldn't be hashed properly, a random hash was used instead. Make sure your transforms and parameters are serializable with pickle or dill for the dataset fingerprinting and caching to work. If you reuse this transform, the caching mechanism will consider it to be different from the previous calls and recompute everything. This warning is only showed once. Subsequent hashing failures won't be showed.\n"
295+
"Parameter 'function'=<function evaluate_stsb at 0x7f92780c7600> of the transform datasets.arrow_dataset.Dataset._map_single couldn't be hashed properly, a random hash was used instead. Make sure your transforms and parameters are serializable with pickle or dill for the dataset fingerprinting and caching to work. If you reuse this transform, the caching mechanism will consider it to be different from the previous calls and recompute everything. This warning is only showed once. Subsequent hashing failures won't be showed.\n"
296296
]
297297
},
298298
{
299299
"data": {
300300
"application/vnd.jupyter.widget-view+json": {
301-
"model_id": "6b39c9e0096a423bbcda949bede6a9cb",
301+
"model_id": "0f28df147f95484c955c3f20f2f954d2",
302302
"version_major": 2,
303303
"version_minor": 0
304304
},
@@ -341,7 +341,7 @@
341341
"output_type": "stream",
342342
"text": [
343343
"vanilla model: pearson= 0.869619439095004\n",
344-
"quantized model: pearson= 0.869619439095004\n",
344+
"quantized model: pearson= 0.869326218489249\n",
345345
"The quantized model achieves 100.0 % accuracy of the fp32 model\n"
346346
]
347347
}
@@ -368,7 +368,7 @@
368368
},
369369
{
370370
"cell_type": "code",
371-
"execution_count": 11,
371+
"execution_count": 7,
372372
"metadata": {},
373373
"outputs": [
374374
{
@@ -391,13 +391,7 @@
391391
"[ INFO ] OpenVINO:\n",
392392
"[ INFO ] Build ................................. 2024.5.0-16971-8a02b4c17bb\n",
393393
"[ INFO ] \n",
394-
"[ INFO ] Device info:\n"
395-
]
396-
},
397-
{
398-
"name": "stdout",
399-
"output_type": "stream",
400-
"text": [
394+
"[ INFO ] Device info:\n",
401395
"[ INFO ] CPU\n",
402396
"[ INFO ] Build ................................. 2024.5.0-16971-8a02b4c17bb\n",
403397
"[ INFO ] \n",
@@ -406,7 +400,7 @@
406400
"[ WARNING ] Performance hint was not explicitly specified in command line. Device(CPU) performance hint will be set to PerformanceMode.LATENCY.\n",
407401
"[Step 4/11] Reading model files\n",
408402
"[ INFO ] Loading model files\n",
409-
"[ INFO ] Read model took 11.28 ms\n",
403+
"[ INFO ] Read model took 10.87 ms\n",
410404
"[ INFO ] Original model I/O parameters:\n",
411405
"[ INFO ] Model inputs:\n",
412406
"[ INFO ] input_ids (node: input_ids) : i64 / [...] / [?,?]\n",
@@ -417,7 +411,7 @@
417411
"[Step 5/11] Resizing model to match image sizes and given batch\n",
418412
"[ INFO ] Model batch size: 1\n",
419413
"[ INFO ] Reshaping model: 'input_ids': [1,384], 'attention_mask': [1,384], 'token_type_ids': [1,384]\n",
420-
"[ INFO ] Reshape model took 2.58 ms\n",
414+
"[ INFO ] Reshape model took 3.02 ms\n",
421415
"[Step 6/11] Configuring input of the model\n",
422416
"[ INFO ] Model inputs:\n",
423417
"[ INFO ] input_ids (node: input_ids) : i64 / [...] / [1,384]\n",
@@ -426,7 +420,7 @@
426420
"[ INFO ] Model outputs:\n",
427421
"[ INFO ] last_hidden_state (node: __module.encoder.layer.5.output.LayerNorm/aten::layer_norm/Add) : f32 / [...] / [1,384,384]\n",
428422
"[Step 7/11] Loading the model to the device\n",
429-
"[ INFO ] Compile model took 132.33 ms\n",
423+
"[ INFO ] Compile model took 125.14 ms\n",
430424
"[Step 8/11] Querying optimal runtime parameters\n",
431425
"[ INFO ] Model:\n",
432426
"[ INFO ] NETWORK_NAME: Model0\n",
@@ -458,17 +452,23 @@
458452
"[ INFO ] Fill input 'token_type_ids' with random values \n",
459453
"[Step 10/11] Measuring performance (Start inference synchronously, limits: 200 iterations)\n",
460454
"[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).\n",
461-
"[ INFO ] First inference took 14.89 ms\n",
455+
"[ INFO ] First inference took 13.97 ms\n"
456+
]
457+
},
458+
{
459+
"name": "stdout",
460+
"output_type": "stream",
461+
"text": [
462462
"[Step 11/11] Dumping statistics report\n",
463463
"[ INFO ] Execution Devices:['CPU']\n",
464464
"[ INFO ] Count: 200 iterations\n",
465-
"[ INFO ] Duration: 2067.30 ms\n",
465+
"[ INFO ] Duration: 1988.82 ms\n",
466466
"[ INFO ] Latency:\n",
467-
"[ INFO ] Median: 9.88 ms\n",
468-
"[ INFO ] Average: 10.15 ms\n",
469-
"[ INFO ] Min: 9.60 ms\n",
470-
"[ INFO ] Max: 11.37 ms\n",
471-
"[ INFO ] Throughput: 96.74 FPS\n"
467+
"[ INFO ] Median: 9.70 ms\n",
468+
"[ INFO ] Average: 9.77 ms\n",
469+
"[ INFO ] Min: 9.54 ms\n",
470+
"[ INFO ] Max: 11.35 ms\n",
471+
"[ INFO ] Throughput: 100.56 FPS\n"
472472
]
473473
}
474474
],
@@ -479,7 +479,7 @@
479479
},
480480
{
481481
"cell_type": "code",
482-
"execution_count": 13,
482+
"execution_count": 8,
483483
"metadata": {},
484484
"outputs": [
485485
{
@@ -502,13 +502,7 @@
502502
"[ INFO ] OpenVINO:\n",
503503
"[ INFO ] Build ................................. 2024.5.0-16971-8a02b4c17bb\n",
504504
"[ INFO ] \n",
505-
"[ INFO ] Device info:\n"
506-
]
507-
},
508-
{
509-
"name": "stdout",
510-
"output_type": "stream",
511-
"text": [
505+
"[ INFO ] Device info:\n",
512506
"[ INFO ] CPU\n",
513507
"[ INFO ] Build ................................. 2024.5.0-16971-8a02b4c17bb\n",
514508
"[ INFO ] \n",
@@ -517,7 +511,7 @@
517511
"[ WARNING ] Performance hint was not explicitly specified in command line. Device(CPU) performance hint will be set to PerformanceMode.LATENCY.\n",
518512
"[Step 4/11] Reading model files\n",
519513
"[ INFO ] Loading model files\n",
520-
"[ INFO ] Read model took 21.99 ms\n",
514+
"[ INFO ] Read model took 15.46 ms\n",
521515
"[ INFO ] Original model I/O parameters:\n",
522516
"[ INFO ] Model inputs:\n",
523517
"[ INFO ] input_ids (node: input_ids) : i64 / [...] / [?,?]\n",
@@ -528,7 +522,7 @@
528522
"[Step 5/11] Resizing model to match image sizes and given batch\n",
529523
"[ INFO ] Model batch size: 1\n",
530524
"[ INFO ] Reshaping model: 'input_ids': [1,384], 'attention_mask': [1,384], 'token_type_ids': [1,384]\n",
531-
"[ INFO ] Reshape model took 3.60 ms\n",
525+
"[ INFO ] Reshape model took 6.89 ms\n",
532526
"[Step 6/11] Configuring input of the model\n",
533527
"[ INFO ] Model inputs:\n",
534528
"[ INFO ] input_ids (node: input_ids) : i64 / [...] / [1,384]\n",
@@ -537,7 +531,7 @@
537531
"[ INFO ] Model outputs:\n",
538532
"[ INFO ] last_hidden_state (node: __module.encoder.layer.5.output.LayerNorm/aten::layer_norm/Add) : f32 / [...] / [1,384,384]\n",
539533
"[Step 7/11] Loading the model to the device\n",
540-
"[ INFO ] Compile model took 324.67 ms\n",
534+
"[ INFO ] Compile model took 325.40 ms\n",
541535
"[Step 8/11] Querying optimal runtime parameters\n",
542536
"[ INFO ] Model:\n",
543537
"[ INFO ] NETWORK_NAME: Model0\n",
@@ -569,17 +563,17 @@
569563
"[ INFO ] Fill input 'token_type_ids' with random values \n",
570564
"[Step 10/11] Measuring performance (Start inference synchronously, limits: 200 iterations)\n",
571565
"[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).\n",
572-
"[ INFO ] First inference took 9.54 ms\n",
566+
"[ INFO ] First inference took 8.49 ms\n",
573567
"[Step 11/11] Dumping statistics report\n",
574568
"[ INFO ] Execution Devices:['CPU']\n",
575569
"[ INFO ] Count: 200 iterations\n",
576-
"[ INFO ] Duration: 906.86 ms\n",
570+
"[ INFO ] Duration: 869.96 ms\n",
577571
"[ INFO ] Latency:\n",
578-
"[ INFO ] Median: 4.19 ms\n",
579-
"[ INFO ] Average: 4.42 ms\n",
580-
"[ INFO ] Min: 4.09 ms\n",
581-
"[ INFO ] Max: 5.56 ms\n",
582-
"[ INFO ] Throughput: 220.54 FPS\n"
572+
"[ INFO ] Median: 4.17 ms\n",
573+
"[ INFO ] Average: 4.23 ms\n",
574+
"[ INFO ] Min: 4.08 ms\n",
575+
"[ INFO ] Max: 6.04 ms\n",
576+
"[ INFO ] Throughput: 229.89 FPS\n"
583577
]
584578
}
585579
],

0 commit comments

Comments
 (0)