ORTOptimizer with bart-large-cnn throwing "Incomplete shape inference"

[ghost]

System Info

NVIDIA Docker container for Pytorch
nvcr.io/nvidia/pytorch:22.10-py3
R9 5900X, RTX A6000, Docker Host is Ubuntu 22.04

Who can help?

@lewtun

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

test_optimum.txt

1. pull down facebook/bart-large-cnn model
2. convert model to ONNX
3. run ORTOptimizer on ONNX model <- fails here

Expected behavior

I expected ORTOptimizer to optimize the model for GPU(sanitize onnx nodes, fold constants, move subgraphs to GPU) but instead it fails with the error: "incomplete symbolic shape inference"

[JingyaHuang]

Hi @dsingal0,

It seems that the symbolic shape infer for some custom fused ops failed while converting float32 to float16. Can you try:

• Firstly, if it works by setting optimization_level=1(without any fusion).
• Then, set optimization_level=2, and disable some fusions to see explicitly which fused op caused the failure.

[JingyaHuang]

Hi @dsingal0, any update about the issue?