Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[TESTS] Make a common entry point for conformance tests #3265

Merged

Conversation

kshpv
Copy link
Collaborator

@kshpv kshpv commented Feb 10, 2025

Changes

Refactors confromance test suite to enhance maintainability and reduce potential bugs.

  1. Unified test entry point
  2. Centralized Fail/XFail logic

Reason for changes

  1. Reduce code complexity
  2. Reduce potential number of bugs

Tests

WC run - https://github.com/openvinotoolkit/nncf/actions/runs/13239414168 - passed
ptq run -job/manual/job/post_training_quantization/616/ - in progress
WC CI - job/manual/job/post_training_weight_compression/317 - passed

@github-actions github-actions bot added the NNCF PTQ Pull requests that updates NNCF PTQ label Feb 10, 2025
@github-actions github-actions bot added the documentation Improvements or additions to documentation label Feb 10, 2025
@kshpv kshpv changed the title [TESTS] Make more common logic in conformance [TESTS] Make a common entry point for conformance tests Feb 10, 2025
@kshpv kshpv marked this pull request as ready for review February 10, 2025 18:19
@kshpv kshpv requested a review from a team as a code owner February 10, 2025 18:19
@kshpv kshpv marked this pull request as draft February 12, 2025 20:25
@kshpv kshpv marked this pull request as ready for review February 12, 2025 22:04
Copy link
Collaborator

@nikita-savelyevv nikita-savelyevv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Activation sparsity conformance test can't be run in the CI because of some infrastructure problems. Could you please run it on the develop and your branches locally and share the results?

@kshpv
Copy link
Collaborator Author

kshpv commented Feb 14, 2025

Local results for sparsity conformance:

PR:

Model Backend Metric name Metric value Metric diff Num int8 Num int4 Num FQ Num sparse activations Compr. time Total time RAM MiB Status Build url
timm/deit3_small_patch16_224_qkv_sparse20_fc1_sparse20_fc2_sparse30 CUDA_TORCH Acc@1 0.8103 -0.0032 0 0 0 36 0:00:16 0:12:10 7370    
timm/deit3_small_patch16_224_qkv_sparse20_fc1_sparse20_fc2_sparse30 TORCH Acc@1 0.8099 -0.0036 0 0 0 36 0:01:07 0:13:03 7707    
tinyllama_ffn_sparse20 CUDA_TORCH Similarity 0.7746 -0.2254 0 0 0 44 0:00:13 0:05:32 2079    
tinyllama_ffn_sparse20 TORCH Similarity 0.79176 -0.20824 0 0 0 44 0:01:30 0:06:52 6218    
tinyllama_int8_asym_data_free_ffn_sparse20 CUDA_TORCH Similarity 0.80373 -0.19627 312 0 0 44 0:00:14 0:02:49 3168    
tinyllama_int8_asym_data_free_ffn_sparse20 TORCH Similarity 0.79003 -0.20997 312 0 0 44 0:01:40 0:04:22 7870    

Develop:

Model Backend Metric name Metric value Metric diff Num FQ Num int4 Num int8 Num sparse activations RAM MiB Compr. time Total time FPS Status
timm/deit3_small_patch16_224_qkv_sparse20_fc1_sparse20_fc2_sparse30 CUDA_TORCH Acc@1 0.8103 -0.0032 0 0 0 36 6313 0:00:15 0:12:17    
timm/deit3_small_patch16_224_qkv_sparse20_fc1_sparse20_fc2_sparse30 TORCH Acc@1 0.8099 -0.0036 0 0 0 36 6544 0:01:07 0:13:04    
tinyllama_ffn_sparse20 CUDA_TORCH Similarity 0.7746 -0.2254 0 0 0 44 2014 0:00:13 0:05:33    
tinyllama_ffn_sparse20 TORCH Similarity 0.79176 -0.20824 0 0 0 44 6294 0:01:30 0:06:53    
tinyllama_int8_asym_data_free_ffn_sparse20 CUDA_TORCH Similarity 0.80373 -0.19627 0 0 312 44 3199 0:00:14 0:02:51    
tinyllama_int8_asym_data_free_ffn_sparse20 TORCH Similarity 0.79003 -0.20997 0 0 312 44 8340 0:01:39 0:04:23    

@AlexanderDokuchaev AlexanderDokuchaev merged commit 033be9b into openvinotoolkit:develop Feb 17, 2025
17 checks passed
AlexanderDokuchaev pushed a commit to AlexanderDokuchaev/nncf that referenced this pull request Feb 17, 2025
…kit#3265)

### Changes
Refactors confromance test suite to enhance maintainability and reduce
potential bugs.

1) Unified test entry point
2) Centralized Fail/XFail logic

### Reason for changes

1) Reduce code complexity
2) Reduce potential number of bugs


### Tests

WC run -
https://github.com/openvinotoolkit/nncf/actions/runs/13239414168 -
passed
ptq run -job/manual/job/post_training_quantization/608/ - in progress
WC CI - job/manual/job/post_training_weight_compression/317 - passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation NNCF PTQ Pull requests that updates NNCF PTQ
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants