OpenVino performances

[Gasp34]

Hi,

I am a member of the DeepFaune and I saw that you are using our model and that you converted it to OpenVino.
Do you have some values of the speed-up it offers ?

[stefanodellosa-personal]

Hi Gasp34,

We're still in early development phase but we're planning to run some dataset based test and some performance measurement soon.

If you're interested we can definitely provide feedback to you once ready, but it might take sometime....

The reason why we're doing such conversion is because as of today WADAS is being optimized for Intel NPU/CPU (but not limited to it). I let @alessandropalla comment here as he is the AI expert.

May I know what's your reference/supported HW to run inference?

As we plan to produce working POC by end of year, please give us a ping something around late December so that we won't forget.

[alessandropalla]

Hi @Gasp34 , we converted DeepFaune to OpenVINO to optimize inference performance of the ViT model on Intel hardware, as our current deployment platform. OpenVINO provides very performant inference for such model especially for latest gen Intel iGPU and NPU. Then we are able to deploy such model on edge servers that are cost-effective while retaining very good inference performance.

Feel free to ask for more info about it. If you are interested to streamline such capabilities directly into DeepFaune I'd be more than happy to help and give guidance

[Gasp34]

Hi to you both,

Thanks for your answers!

I was interested to include OpenVino directly into the DeepFaune software as I guess that most of the users only have a CPU and probably an Intel one. I will try on my machine to see if there is an improved inference performance!

[alessandropalla]

@Gasp34 , you can use this class we implemented in this repo as a drop-in replacement of the DeepFaune one. The expectation is for openvino to be faster on intel hw than vanilla pytorch. Also consider that depending on your application inference can be optimized for latency/throughput and if you have a reference dataset, the model can be quantized to get extra performance out with hopefully little accuracy impact

Here some pointers:

• Performance tuning link
• Post-training quantization link

[Gasp34]

Thank you very much !
Post-training quantization seems very interesting !

Btw, maybe you haven't seen yet, but we have released an updated version of the model, with 4 new classes.

[alessandropalla]

Is deepfaune-vit_large_patch14_dinov2.lvd142m.v2.pt ? I'd love to try it out

By the way, do you have the training/testing dataset you used to train publicly available? I can give it a try to do accuracy aware quantization on it with the new model

[Gasp34]

Yes it is deepfaune-vit_large_patch14_dinov2.lvd142m.v2.pt that you can find here.

Sadly most of the training dataset doesn't belong to us, so I cannot share it. However on these two Zenodo repo you can find some cropped image (more than 200K I believe) that we used for training : https://zenodo.org/records/10925926, https://zenodo.org/records/10014376

[stefanodellosa-personal]

Hi @Gasp34 ,

Just to inform you that we're having some material about miss of the classification model that might be interesting/useful for you.

If this is the case, you could share your contact through our website (https://www.wadas.it/en/contact-us/) so we can plan how to enable this excange of info.

Regards,
Stefano