Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CI] [GHA] Transfer Linux CUDA workflow from Azure to GHA #909

Merged
merged 41 commits into from
May 22, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
276ddc8
transfer linux workflow
akashchi Apr 15, 2024
9fb3bd6
provide ref
akashchi Apr 15, 2024
1b56f83
install git-lfs
akashchi Apr 15, 2024
f62d347
install unzip
akashchi Apr 15, 2024
b4de031
correct path for log
akashchi Apr 15, 2024
eceb039
upload test results
akashchi Apr 15, 2024
715be12
use gha runner
akashchi Apr 17, 2024
f46dfb8
rm sudo
akashchi Apr 17, 2024
f64b8ae
skip test
akashchi Apr 19, 2024
5799a96
rm unused env vars from linux; add win workflow
akashchi Apr 22, 2024
0ee1669
configure msvc
akashchi Apr 22, 2024
1ac87a3
add build of ov_contrib
akashchi Apr 22, 2024
838d87a
use setupvars
akashchi Apr 22, 2024
371de19
use quotes
akashchi Apr 22, 2024
ee7a6fa
use ov repo
akashchi Apr 22, 2024
52883c3
use cmd
akashchi Apr 22, 2024
f04b96e
call cmd setupvars
akashchi Apr 23, 2024
c2e081d
add nvidia job
akashchi Apr 24, 2024
e293f8d
add generator
akashchi Apr 24, 2024
61cb075
do not use dev package
akashchi Apr 24, 2024
8f635dc
add missing ref
akashchi Apr 24, 2024
2aac984
allow downgrades
akashchi Apr 24, 2024
208fba1
allow change
akashchi Apr 24, 2024
bb7a738
add dev package
akashchi Apr 25, 2024
dc55ff9
enable system
akashchi Apr 25, 2024
4d28ac1
use from fork
akashchi Apr 25, 2024
c79b8ca
rm system
akashchi Apr 25, 2024
7316b6d
use hash
akashchi Apr 25, 2024
ebe6d32
update hash
akashchi Apr 25, 2024
e6ab978
enable tests
akashchi Apr 25, 2024
689d68a
build dev targets, do not build tests
akashchi Apr 26, 2024
1a41376
enable tests
akashchi Apr 26, 2024
5eb5437
use master
akashchi Apr 29, 2024
455c5bd
rm unused
akashchi May 3, 2024
34c3f82
use actions/cache for ccache
akashchi May 16, 2024
8400f50
unskip
akashchi May 16, 2024
87889c3
Merge remote-tracking branch 'upstream/master' into ci/linux-workflow
akashchi May 16, 2024
f65b62b
merge
akashchi May 17, 2024
17955cb
use actions/cache in nvidia job
akashchi May 17, 2024
f119a0b
return package dir
akashchi May 17, 2024
c56e1ca
merge and disable NPU
akashchi May 22, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
158 changes: 157 additions & 1 deletion .github/workflows/linux.yml
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@ jobs:
TEST_DATA: /__w/openvino_contrib/openvino_contrib/testdata
INSTALL_DIR: /__w/openvino_contrib/openvino_contrib/openvino_install
BUILD_DIR: /__w/openvino_contrib/openvino_contrib/openvino_build
DEVELOPER_PACKAGE_DIR: /__w/openvino_contrib/openvino_contrib/developer_package_install
CCACHE_DIR: /__w/openvino_contrib/openvino_contrib/ccache
CCACHE_MAXSIZE: "2G"
GRADLE_VER: '7.1.1'
Expand Down Expand Up @@ -129,7 +130,10 @@ jobs:
-DENABLE_OV_PYTORCH_FRONTEND=OFF \
-DOPENVINO_EXTRA_MODULES=${OPENVINO_CONTRIB_REPO}/modules \
-DENABLE_PYTHON=ON \
-DCPACK_GENERATOR=TGZ \
-DENABLE_WHEEL=ON \
-DENABLE_TESTS=ON \
-DENABLE_INTEL_NPU=OFF \
-DCMAKE_CXX_COMPILER_LAUNCHER=${{ env.CMAKE_CXX_COMPILER_LAUNCHER }} \
-DCMAKE_C_COMPILER_LAUNCHER=${{ env.CMAKE_C_COMPILER_LAUNCHER }} \
-S ${OPENVINO_REPO} \
Expand All @@ -145,7 +149,9 @@ jobs:
run: ccache --show-stats

- name: Cmake install
run: cmake -DCMAKE_INSTALL_PREFIX=${INSTALL_DIR} -P ${BUILD_DIR}/cmake_install.cmake
run: |
cmake -DCMAKE_INSTALL_PREFIX=${INSTALL_DIR} -P ${BUILD_DIR}/cmake_install.cmake
cmake -DCMAKE_INSTALL_PREFIX=${DEVELOPER_PACKAGE_DIR} -DCOMPONENT=developer_package -P ${BUILD_DIR}/cmake_install.cmake

- name: Java tests
working-directory: ${{ env.OPENVINO_CONTRIB_REPO }}/modules/java_api
Expand All @@ -168,10 +174,160 @@ jobs:
env:
CUSTOM_OP_LIB: ${{ env.OPENVINO_REPO }}/bin/intel64/${{ env.CMAKE_BUILD_TYPE }}/libuser_ov_extensions.so

- name: Pack Artifacts
run: |
pushd ${INSTALL_DIR}
tar -czvf ${BUILD_DIR}/openvino_package.tar.gz *
popd

pushd ${DEVELOPER_PACKAGE_DIR}
tar -czvf ${BUILD_DIR}/openvino_developer_package.tar.gz *
popd

- name: Upload Test Results
uses: actions/upload-artifact@v4
if: ${{ !cancelled() }}
with:
name: test-results-java
path: ${{ env.OPENVINO_CONTRIB_REPO }}/modules/java_api/build/test-results
if-no-files-found: 'warn'

- name: Upload openvino package
if: ${{ always() }}
uses: actions/upload-artifact@v4
with:
name: openvino_package
path: ${{ env.BUILD_DIR }}/openvino_package.tar.gz
if-no-files-found: 'error'

- name: Upload openvino developer package
if: ${{ always() }}
uses: actions/upload-artifact@v4
with:
name: openvino_developer_package
path: ${{ env.BUILD_DIR }}/openvino_developer_package.tar.gz
if-no-files-found: 'error'

NVIDIA_Plugin:
name: NVIDIA plugin
needs: Build_and_test
timeout-minutes: 40
defaults:
run:
shell: bash
runs-on: ubuntu-20.04-16-cores
container:
image: nvidia/cuda:11.8.0-cudnn8-runtime-ubuntu20.04
env:
CMAKE_BUILD_TYPE: 'Release'
CMAKE_GENERATOR: 'Ninja Multi-Config'
CMAKE_CUDA_COMPILER_LAUNCHER: ccache
CMAKE_CXX_COMPILER_LAUNCHER: ccache
CMAKE_C_COMPILER_LAUNCHER: ccache
INSTALL_DIR: /__w/openvino_contrib/openvino_contrib/install
OPENVINO_DEVELOPER_PACKAGE: /__w/openvino_contrib/openvino_contrib/install/developer_package
OPENVINO_REPO: /__w/openvino_contrib/openvino_contrib/openvino
OPENVINO_CONTRIB_REPO: /__w/openvino_contrib/openvino_contrib/openvino_contrib
NVIDIA_BUILD_DIR: /__w/openvino_contrib/openvino_contrib/nvidia_plugin_build
CCACHE_DIR: /__w/openvino_contrib/openvino_contrib/ccache
CCACHE_MAXSIZE: "2G"
DEBIAN_FRONTEND: 'noninteractive'

steps:
- name: Set apt retries
run: echo 'Acquire::Retries "10";' > /etc/apt/apt.conf.d/80-retries

- name: Fetch install_build_dependencies.sh
uses: actions/checkout@v4
with:
sparse-checkout: |
install_build_dependencies.sh
sparse-checkout-cone-mode: false
repository: 'openvinotoolkit/openvino'
ref: 'master'
path: ${{ env.OPENVINO_REPO }}

- name: Install Prerequisites
run: apt update && apt install -y git ca-certificates

- name: Download OpenVINO package
uses: actions/download-artifact@v4
with:
name: openvino_package
path: ${{ env.INSTALL_DIR }}

- name: Download OpenVINO Developer package
uses: actions/download-artifact@v4
with:
name: openvino_developer_package
path: ${{ env.INSTALL_DIR }}

- name: Extract OpenVINO packages
run: |
pushd ${INSTALL_DIR}
tar -xzf openvino_package.tar.gz -C ${INSTALL_DIR}
popd

pushd ${INSTALL_DIR}
tar -xzf openvino_developer_package.tar.gz -C ${INSTALL_DIR}
popd

- name: Clone OpenVINO Contrib
uses: actions/checkout@v4
with:
path: ${{ env.OPENVINO_CONTRIB_REPO }}

#
# Dependencies
#

- name: Install build dependencies
run: |
${OPENVINO_REPO}/install_build_dependencies.sh
apt -y --no-install-recommends install software-properties-common curl

- name: Setup ccache
uses: actions/cache@v4
with:
# Should save cache only if run in the master branch of the base repo
# github.ref_name is 'ref/PR_#' in case of the PR, and 'branch_name' when executed on push
save-always: ${{ github.ref_name == 'master' && 'true' || 'false' }}
path: ${{ env.CCACHE_DIR }}
key: ${{ runner.os }}-${{ runner.arch }}-nvidia-ccache-${{ github.sha }}
restore-keys: |
${{ runner.os }}-${{ runner.arch }}-nvidia-ccache

- name: Install CUDA
run: |
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-ubuntu2004.pin
mv cuda-ubuntu2004.pin /etc/apt/preferences.d/cuda-repository-pin-600

apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/3bf863cc.pub
add-apt-repository "deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/ /"
apt update
apt install -y --allow-downgrades --allow-change-held-packages \
libcudnn8=8.9.4.*-1+cuda11.8 \
libcudnn8-dev=8.9.4.*-1+cuda11.8 \
libcudnn8-samples=8.9.4.*-1+cuda11.8 \
cuda-runtime-11-8 \
cuda-11-8 \
libcutensor1=1.6.1.5-1 \
libcutensor-dev=1.6.1.5-1 \
cuda-drivers=520.61.05-1

#
# Build
#

- name: Cmake & Build - NVIDIA Plugin
run: |
source ${INSTALL_DIR}/setupvars.sh
cmake \
-DOpenVINODeveloperPackage_DIR=${OPENVINO_DEVELOPER_PACKAGE}/cmake \
-DCMAKE_COMPILE_WARNING_AS_ERROR=OFF \
-S ${OPENVINO_CONTRIB_REPO}/modules/nvidia_plugin \
-B ${NVIDIA_BUILD_DIR}
cmake --build ${NVIDIA_BUILD_DIR} --parallel --config ${{ env.CMAKE_BUILD_TYPE }} --verbose -- ov_nvidia_func_tests ov_nvidia_unit_tests

- name: Show ccache stats
run: ccache --show-stats
1 change: 1 addition & 0 deletions .github/workflows/windows.yml
Original file line number Diff line number Diff line change
Expand Up @@ -132,6 +132,7 @@ jobs:
-DENABLE_CPPLINT=OFF `
-DENABLE_SAMPLES=OFF `
-DENABLE_PYTHON=ON `
-DENABLE_INTEL_NPU=OFF `
-DENABLE_JS=OFF `
-DOPENVINO_EXTRA_MODULES=${{ env.OPENVINO_CONTRIB_REPO }}/modules `
-DCMAKE_CXX_COMPILER_LAUNCHER=${{ env.CMAKE_CXX_COMPILER_LAUNCHER }} `
Expand Down
Loading