Update ONNX_HUB_MAINIFEST with the latest files and add CI check for it (#547)

* enable partial manifest generation

Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com>

* cd workflow_scripts

Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com>

* python -m pip install typepy BeautifulSoup4

Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com>

* pip install

Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com>

* reverse

Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com>

* git-lfs if not exist

Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com>

* change PATH to make it exec under root dir

Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com>

* use --drop instead of --keep

Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com>

* --drop

Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com>

* Only git-lfs when needed

Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com>

Author: jcwchen

.github/PULL_REQUEST_TEMPLATE.md

@@ -128,6 +128,9 @@ https://github.com/microsoft/onnxruntime/blob/master/tools/python/PythonTools.md
 
 <hr>
 
+### Update ONNX_HUB_MANIFEST.json for ONNX Hub
+If this PR does update/add .onnx or .tar.gz files, please use `python workflow_scripts/generate_onnx_hub_manifest.py --target diff` to update ONNX_HUB_MANIFEST.json with according model information (especially SHA) for ONNX Hub.
+
 ### References
 Link to paper or references.
 

.github/workflows/linux_ci.yml

@@ -33,17 +33,19 @@ jobs:
       - name: Install dependencies
         run: |
           python -m pip install --upgrade pip
-          # TODO: now ONNX only supports Protobuf <= 3.20.1
-          python -m pip install protobuf==3.20.1
           python -m pip install onnx onnxruntime requests py-cpuinfo
           python -m cpuinfo
 
+      - name: Test updated ONNX_HUB_MANIFEST.json
+        run: |
+          python -m pip install typepy BeautifulSoup4 markdown pandas
+          python workflow_scripts/generate_onnx_hub_manifest.py --target diff --drop
+          git diff --exit-code -- ONNX_HUB_MANIFEST.json || { echo 'Please use "python workflow_scripts/generate_onnx_hub_manifest.py --target diff" to update ONNX_HUB_MANIFEST.json.' ; exit 1; }
+
       - name: Test new models by onnx
         run: |
-          python workflow_scripts/test_models.py --target onnx
+          python workflow_scripts/test_models.py --target onnx --drop
 
       - name: Test new models by onnxruntime
         run: |
-          python workflow_scripts/test_models.py --target onnxruntime
-
-    
+          python workflow_scripts/test_models.py --target onnxruntime --drop

.github/workflows/windows_ci.yml

@@ -40,8 +40,8 @@ jobs:
 
       - name: Test new models by onnx
         run: |
-          python workflow_scripts/test_models.py --target onnx
+          python workflow_scripts/test_models.py --target onnx --drop
 
       - name: Test new models by onnxruntime
         run: |
-          python workflow_scripts/test_models.py --target onnxruntime
+          python workflow_scripts/test_models.py --target onnxruntime --drop