Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[GHA] Save artifacts to cloud share #1943

Open
wants to merge 20 commits into
base: master
Choose a base branch
from
+172 −5
Open

[GHA] Save artifacts to cloud share #1943

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
be12544
[GHA] Save Linux artifacts to cloud share
akladiev Mar 19, 2025
c848183
[GHA] Save Windows artifacts to cloud share
akladiev Mar 19, 2025
8905b6b
Add missing step
akladiev Mar 19, 2025
1ca5336
Fix build-type
akladiev Mar 19, 2025
ad4fbf1
Fixes for cpack
akladiev Mar 19, 2025
f517b88
Fixes for wheels
akladiev Mar 19, 2025
1002000
Merge branch 'master' into save_artifacts_cloud_share
akladiev Mar 23, 2025
06049ec
Align structure with tokenizers
akladiev Mar 23, 2025
e93d717
Merge branch 'save_artifacts_cloud_share' of https://github.com/aklad…
akladiev Mar 23, 2025
0b7311d
Update linux.yml
akladiev Mar 23, 2025
f0fb4fb
Revert "Update linux.yml"
akladiev Mar 23, 2025
14ebd7d
Revert "Align structure with tokenizers"
akladiev Mar 23, 2025
f02dc0c
Use separate artifacts folder structure
akladiev Mar 23, 2025
ebdd5ee
Add extracting cpack archive on Windows
akladiev Mar 24, 2025
2df629e
Remove extra steps
akladiev Mar 24, 2025
25cdae7
Fix archive name
akladiev Mar 24, 2025
93d096f
Merge branch 'master' into save_artifacts_cloud_share
akladiev Mar 25, 2025
88ecd0c
Update .github/workflows/linux.yml
akladiev Mar 25, 2025
52b2a9e
Use action from master
akladiev Mar 25, 2025
da1858d
Update .github/workflows/linux.yml
akladiev Mar 25, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
79 changes: 78 additions & 1 deletion .github/workflows/linux.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,10 @@ env:
HF_HOME: /mount/caches/huggingface
GENAI_ARCHIVE_NAME: genai.tar.gz
GENAI_SAMPLES_NAME: genai_samples.tar.gz
ARTIFACTS_SHARE: '/mount/build-artifacts'
BASE_PRODUCT_TYPE: public_linux_ubuntu_22_04_x86_64
GENAI_WHEELS_ARTIFACT_NAME: 'genai_wheels'
GENAI_ARCHIVE_ARTIFACT_BASE_NAME: 'genai_archive'

jobs:
openvino_download:
Expand Down Expand Up @@ -97,6 +101,7 @@ jobs:
INSTALL_TESTS_DIR: ${{ github.workspace }}/tests
BUILD_DIR: ${{ github.workspace }}/build
SRC_DIR: ${{ github.workspace }}/src
MANIFEST_PATH: ${{ github.workspace }}/manifest.yml

steps:
- name: Clone openvino.genai
Expand All @@ -105,6 +110,17 @@ jobs:
path: ${{ env.SRC_DIR }}
submodules: recursive

- name: Generate product manifest
id: create_manifest
uses: openvinotoolkit/openvino/.github/actions/create_manifest@master
with:
repos: |
${{ env.SRC_DIR }}
product_type: ${{ env.BASE_PRODUCT_TYPE }}_${{ matrix.build-type }}
target_arch: 'x86_64'
build_type: ${{ matrix.build-type }}
save_to: ${{ env.MANIFEST_PATH }}

- name: Download OpenVINO package
uses: actions/download-artifact@fa0a91b85d4f404e444e00e005971372dc801d16 # v4.1.8
with:
Expand Down Expand Up @@ -149,6 +165,14 @@ jobs:
path: ${{ env.INSTALL_TESTS_DIR }}
if-no-files-found: 'error'

- name: Upload manifest
if: ${{ always() }}
uses: actions/upload-artifact@4cec3d8aa04e39d1a68397de0c4cd6fb9dce8ec1 # v4.6.1
with:
name: manifest_${{ matrix.build-type }}
path: ${{ env.MANIFEST_PATH }}
if-no-files-found: 'error'

genai_build_wheel:
name: Build Wheel
needs: [ openvino_download ]
Expand Down Expand Up @@ -208,10 +232,63 @@ jobs:
if: ${{ always() }}
uses: actions/upload-artifact@b4b15b8c7c6ac21ea08fcf65892d2ee8f75cf882 # v4.4.3
with:
name: genai_wheels
name: ${{ env.GENAI_WHEELS_ARTIFACT_NAME }}
path: ${{ env.INSTALL_DIR }}
if-no-files-found: 'error'

store_artifacts:
name: Store build artifacts
strategy:
matrix:
build-type: [Release]
needs: [openvino_download, genai_build_wheel, genai_build_cmake]
timeout-minutes: 10
defaults:
run:
shell: bash
runs-on: aks-linux-medium
container:
image: openvinogithubactions.azurecr.io/library/python:3.12-slim
volumes:
- /mount:/mount
- ${{ github.workspace }}:${{ github.workspace }}
env:
CPACK_PATH: ${{ github.workspace }}/ov_genai
CPACK_PACKAGE: ${{ github.workspace }}/ov_genai.tar.gz
WHEEL_PACKAGE: ${{ github.workspace }}/wheels
MANIFEST_PATH: ${{ github.workspace }}/manifest.yml

steps:
- name: Download genai package
uses: actions/download-artifact@cc203385981b70ca67e1cc392babf9cc229d5806 # v4.1.9
with:
name: ${{ env.GENAI_ARCHIVE_ARTIFACT_BASE_NAME }}_${{ matrix.build-type }}
path: ${{ env.CPACK_PATH }}

- name: Pack Artifacts
run: tar -czvf ${{ env.CPACK_PACKAGE }} *
working-directory: ${{ env.CPACK_PATH }}

- name: Download manifest and wheels
uses: actions/download-artifact@cc203385981b70ca67e1cc392babf9cc229d5806 # v4.1.9
with:
pattern: '{${{ env.GENAI_WHEELS_ARTIFACT_NAME }},manifest_${{ matrix.build-type }}}'
path: ${{ github.workspace }}
merge-multiple: true

- name: Store ${{ matrix.build_type }} artifacts to a shared drive
id: store_artifacts
if: ${{ always() }}
uses: openvinotoolkit/openvino/.github/actions/store_artifacts@master
with:
artifacts: |
${{ env.CPACK_PACKAGE }}
${{ env.WHEEL_PACKAGE }}
${{ env.MANIFEST_PATH }}
storage_dir: ${{ env.BASE_PRODUCT_TYPE }}_${{ matrix.build-type }}
storage_root: ${{ env.ARTIFACTS_SHARE }}
product_name: ${{ github.event.repository.name }}

genai_build_samples:
name: Build Samples - ${{ matrix.build-type }}
strategy:
Expand Down
98 changes: 94 additions & 4 deletions .github/workflows/windows.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,10 @@ env:
CMAKE_CXX_COMPILER_LAUNCHER: ccache
CMAKE_C_COMPILER_LAUNCHER: ccache
CCACHE_MAXSIZE: 500Mi
ARTIFACTS_SHARE: '/mount/build-artifacts'
BASE_PRODUCT_TYPE: public_windows_vs2022
GENAI_WHEELS_ARTIFACT_NAME: 'genai_wheels'
GENAI_ARCHIVE_ARTIFACT_BASE_NAME: 'genai_cpack'

jobs:
openvino_download:
Expand Down Expand Up @@ -70,6 +74,7 @@ jobs:
SRC_DIR: ${{ github.workspace }}\src\genai
BUILD_DIR: ${{ github.workspace }}\build\genai
CCACHE_DIR: ${{ github.workspace }}\ccache
MANIFEST_PATH: ${{ github.workspace }}\manifest.yml

steps:
- name: Clone genai
Expand All @@ -83,6 +88,17 @@ jobs:
with:
python-version: ${{ env.PYTHON_VERSION }}

- name: Generate product manifest
id: create_manifest
uses: openvinotoolkit/openvino/.github/actions/create_manifest@master
with:
repos: |
${{ env.SRC_DIR }}
product_type: ${{ env.BASE_PRODUCT_TYPE }}_${{ matrix.build-type }}
target_arch: 'x86_64'
build_type: ${{ matrix.build-type }}
save_to: ${{ env.MANIFEST_PATH }}

- name: Download OpenVINO package
uses: actions/download-artifact@fa0a91b85d4f404e444e00e005971372dc801d16 # v4.1.8
with:
Expand Down Expand Up @@ -143,6 +159,16 @@ jobs:
#
# Upload build artifacts
#

- name: Pack Artifacts
run: |
$file=Get-ChildItem -Path "${{ env.GENAI_INSTALL_DIR }}"
$compress = @{
Path = $file
CompressionLevel = "Optimal"
DestinationPath = "${{ env.BUILD_DIR }}/${{ env.GENAI_ARCHIVE_ARTIFACT_BASE_NAME }}.zip"
}
Compress-Archive @compress

- name: Save ccache
if: always() && steps.ccache-restore.outputs.cache-hit != 'true' && github.event_name == 'push'
Expand All @@ -156,24 +182,32 @@ jobs:
uses: actions/upload-artifact@65c4c4a1ddee5b72f698fdd19549f0f0fb45cf08 # v4.6.0
with:
name: genai_cpack_${{ matrix.build-type }}
path: ${{ env.GENAI_INSTALL_DIR }}
path: ${{ env.BUILD_DIR }}/*.zip
if-no-files-found: 'error'

- name: Upload Tools
if: always()
uses: actions/upload-artifact@b4b15b8c7c6ac21ea08fcf65892d2ee8f75cf882 # v4.4.3
with:
name: genai_tools_${{ matrix.build-type }}
path: ${{ env.INSTALL_TOOLS_DIR }}
if-no-files-found: 'error'

- name: Upload Tests
if: always()
uses: actions/upload-artifact@b4b15b8c7c6ac21ea08fcf65892d2ee8f75cf882 # v4.4.3
with:
name: genai_tests_${{ matrix.build-type }}
path: ${{ env.INSTALL_TESTS_DIR }}
if-no-files-found: 'error'

- name: Upload manifest
if: ${{ always() }}
uses: actions/upload-artifact@4cec3d8aa04e39d1a68397de0c4cd6fb9dce8ec1 # v4.6.1
with:
name: manifest_${{ matrix.build-type }}
path: ${{ env.MANIFEST_PATH }}
if-no-files-found: 'error'

genai_build_wheel:
name: genai wheel
Expand Down Expand Up @@ -280,6 +314,53 @@ jobs:
path: ${{ env.INSTALL_DIR }}
if-no-files-found: 'error'

store_artifacts:
name: Store build artifacts
strategy:
matrix:
build-type: [Release]
needs: [openvino_download, genai_build_wheel, genai_build_cpack]
timeout-minutes: 10
defaults:
run:
shell: bash
runs-on: aks-linux-2-cores-8gb
container:
image: openvinogithubactions.azurecr.io/library/python:3.12-slim
volumes:
- /mount:/mount
- ${{ github.workspace }}:${{ github.workspace }}
env:
WHEEL_PACKAGE: ${{ github.workspace }}/wheels
MANIFEST_PATH: ${{ github.workspace }}/manifest.yml

steps:
- name: Download genai package
uses: actions/download-artifact@cc203385981b70ca67e1cc392babf9cc229d5806 # v4.1.9
with:
name: ${{ env.GENAI_ARCHIVE_ARTIFACT_BASE_NAME }}_${{ matrix.build-type }}
path: ${{ github.workspace }}

- name: Download manifest and wheels
uses: actions/download-artifact@cc203385981b70ca67e1cc392babf9cc229d5806 # v4.1.9
with:
pattern: '{${{ env.GENAI_WHEELS_ARTIFACT_NAME }},manifest_${{ matrix.build-type }}}'
path: ${{ github.workspace }}
merge-multiple: true

- name: Store ${{ matrix.build_type }} artifacts to a shared drive
id: store_artifacts
if: ${{ always() }}
uses: openvinotoolkit/openvino/.github/actions/store_artifacts@master
with:
artifacts: |
${{ github.workspace }}/${{ env.GENAI_ARCHIVE_ARTIFACT_BASE_NAME }}.zip
${{ env.WHEEL_PACKAGE }}
${{ env.MANIFEST_PATH }}
storage_dir: ${{ env.BASE_PRODUCT_TYPE }}_${{ matrix.build-type }}
storage_root: ${{ env.ARTIFACTS_SHARE }}
product_name: ${{ github.event.repository.name }}

genai_build_samples:
name: Build Samples - ${{ matrix.build-type }}
strategy:
Expand Down Expand Up @@ -311,6 +392,9 @@ jobs:
path: ${{ env.OV_INSTALL_DIR }}
merge-multiple: true

- name: Extract Artifacts
run: Expand-Archive -Path ${{ env.OV_INSTALL_DIR }}/${{ env.GENAI_ARCHIVE_ARTIFACT_BASE_NAME }}.zip -DestinationPath ${{ env.OV_INSTALL_DIR }}

- name: Build Samples (Release)
if: ${{ 'Release' == matrix.build-type }}
run: |
Expand Down Expand Up @@ -500,6 +584,9 @@ jobs:
path: ${{ env.INSTALL_DIR }}
merge-multiple: true

- name: Extract Artifacts
run: Expand-Archive -Path ${{ env.INSTALL_DIR }}/${{ env.GENAI_ARCHIVE_ARTIFACT_BASE_NAME }}.zip -DestinationPath ${{ env.INSTALL_DIR }}

- name: Download GenAI JS Bildings Artifacts
uses: actions/download-artifact@fa0a91b85d4f404e444e00e005971372dc801d16 # v4.1.8
with:
Expand Down Expand Up @@ -582,7 +669,10 @@ jobs:
pattern: "{${{ needs.openvino_download.outputs.ov_artifact_name }},genai_cpack_${{ matrix.build-type }},genai_tools_${{ matrix.build-type }},genai_tests_${{ matrix.build-type }},genai_wheels}"
path: ${{ env.INSTALL_DIR }}
merge-multiple: true


- name: Extract Artifacts
run: Expand-Archive -Path ${{ env.INSTALL_DIR }}/${{ env.GENAI_ARCHIVE_ARTIFACT_BASE_NAME }}.zip -DestinationPath ${{ env.INSTALL_DIR }}

- name: Setup Python ${{ env.PYTHON_VERSION }}
uses: actions/setup-python@42375524e23c412d93fb67b49958b491fce71c38 # v5.4.0
with:
Expand Down
Loading