Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use software.repos.intel.com Conda Channel #236

Merged
merged 6 commits into from
Jul 12, 2024
Merged

Use software.repos.intel.com Conda Channel #236

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
efc624c
switch from anaconda to intel software repo for conda
Jul 12, 2024
1e87e27
Update Dockerfile
Jul 12, 2024
208eed1
Merge branch 'main' into tylertitsworth/intel-conda-channel
Jul 12, 2024
ee74ff9
modify intel channel var
Jul 12, 2024
5ef01e9
remove duplicate channel
Jul 12, 2024
04ee965
Fetch dataset-librarian with pip
Jul 12, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions preset/classical-ml/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ RUN wget --progress=dot:giga --no-check-certificate https://github.com/conda-for
export PATH="${CONDA_ROOT}/bin/:${PATH}" && \
conda update -y conda && \
conda config --add channels conda-forge && \
conda config --add channels intel && \
conda config --add channels https://software.repos.intel.com/python/conda/ && \
conda init --all && \
conda install -y \
'jupyterlab>=4.1.8' \
Expand Down Expand Up @@ -102,7 +102,6 @@ RUN conda create -yn classical-ml -c ${INTEL_CHANNEL} -c conda-forge \
'python-dotenv>=1.0.1' \
'tqdm>=4.66.2' \
'matplotlib-base>=3.4.3' \
'dataset_librarian>=1.0.4' \
'threadpoolctl>=3.3.0' \
'ipython>=8.18.1' \
'ipykernel>=6.29.3' \
Expand All @@ -116,6 +115,7 @@ RUN conda create -yn classical-ml -c ${INTEL_CHANNEL} -c conda-forge \

# PyPI packages
RUN conda run -n classical-ml python -m pip install --no-deps --no-cache-dir \
'dataset-librarian==1.0.4' \
'cloud-data-connector==1.0.3'


Expand Down
2 changes: 1 addition & 1 deletion preset/classical-ml/docker-compose.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ services:
BASE_TAG: ${BASE_TAG:-22.04}
DPNP_VERSION: ${NUMBA_DPEX_VERSION:-0.14.0}
IDP_VERSION: ${IDP_VERSION:-2024.1.0}
INTEL_CHANNEL: ${INTEL_CHANNEL:-intel}
INTEL_CHANNEL: ${INTEL_CHANNEL:-https://software.repos.intel.com/python/conda/}
MINIFORGE_VERSION: ${MINIFORGE_VERSION:-Linux-x86_64}
MODIN_VERSION: ${MODIN_VERSION:-0.26.1}
MPI_VERSION: ${MPI_VERSION:-2021.12.0}
Expand Down
4 changes: 2 additions & 2 deletions preset/data-analytics/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ RUN wget --progress=dot:giga --no-check-certificate "https://github.com/conda-fo
export PATH="${CONDA_ROOT}/bin/:${PATH}" && \
conda update -y conda && \
conda config --add channels conda-forge && \
conda config --add channels intel && \
conda config --add channels https://software.repos.intel.com/python/conda/ && \
conda init --all && \
conda install -y \
'jupyterlab>=4.1.8' \
Expand Down Expand Up @@ -98,7 +98,6 @@ RUN conda create -yn data-analytics -c "${INTEL_CHANNEL}" -c conda-forge \
'python-dotenv>=1.0.1' \
'tqdm>=4.66.2' \
'matplotlib-base>=3.4.3' \
'dataset_librarian>=1.0.4' \
'threadpoolctl>=3.3.0' \
'ipython>=8.18.1' \
'ipykernel>=6.29.3' \
Expand All @@ -110,6 +109,7 @@ RUN conda create -yn data-analytics -c "${INTEL_CHANNEL}" -c conda-forge \
conda clean -y --all

RUN conda run -n data-analytics python -m pip install --no-deps --no-cache-dir \
'dataset-librarian==1.0.4' \
'cloud-data-connector==1.0.3'

FROM data-analytics-python as data-analytics-jupyter
Expand Down
2 changes: 1 addition & 1 deletion preset/data-analytics/docker-compose.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ services:
BASE_TAG: ${BASE_TAG:-22.04}
DPNP_VERSION: ${NUMBA_DPEX_VERSION:-0.14.0}
IDP_VERSION: ${IDP_VERSION:-2024.1.0}
INTEL_CHANNEL: ${INTEL_CHANNEL:-intel}
INTEL_CHANNEL: ${INTEL_CHANNEL:-https://software.repos.intel.com/python/conda/}
MINIFORGE_VERSION: ${MINIFORGE_VERSION:-Linux-x86_64}
MODIN_VERSION: ${MODIN_VERSION:-0.26.1}
MPI_VERSION: ${MPI_VERSION:-2021.12.0}
Expand Down
24 changes: 12 additions & 12 deletions preset/deep-learning/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -137,7 +137,7 @@ RUN wget --progress=dot:giga --no-check-certificate "https://github.com/conda-fo
export PATH="${CONDA_ROOT}/bin/:${PATH}" && \
conda update -y conda && \
conda config --add channels conda-forge && \
conda config --add channels intel && \
conda config --add channels https://software.repos.intel.com/python/conda/ && \
conda init --all && \
conda install -c conda-forge \
'jupyterlab>=4.1.8' \
Expand Down Expand Up @@ -172,7 +172,7 @@ ARG TORCHAUDIO_CPU_VERSION
ARG DEEPSPEED_VERSION

# PyTorch CPU Env - conda packages
RUN conda create -yn pytorch-cpu -c intel/label/oneapi -c "${INTEL_CHANNEL}" -c conda-forge \
RUN conda create -yn pytorch-cpu -c "${INTEL_CHANNEL}" -c conda-forge \
dpnp="${DPNP_VERSION}" \
numpy="${NUMPY_VERSION}" \
python="${PYTHON_VERSION}" \
Expand All @@ -199,8 +199,8 @@ RUN conda run -n pytorch-cpu pip install --no-deps --no-cache-dir --ignore-insta
'ninja>=1.11.1.1' \
'python-dotenv>=1.0.1' \
'tqdm>=4.66.2' \
'cloud-data-connector>=1.0.3' \
'dataset-librarian>=1.0.4' && \
'cloud-data-connector==1.0.3' \
'dataset-librarian==1.0.4' && \
conda run -n pytorch-cpu pip install --no-cache-dir --ignore-installed \
'transformers>=4.40.2' \
'datasets>=2.19.1' \
Expand All @@ -227,7 +227,7 @@ ARG IDEX_VERSION
ARG DEEPSPEED_VERSION

# PyTorch GPU Env - conda packages
RUN conda create -yn pytorch-gpu -c intel/label/oneapi -c "${INTEL_CHANNEL}" -c conda-forge \
RUN conda create -yn pytorch-gpu -c "${INTEL_CHANNEL}" -c conda-forge \
dpnp="${DPNP_VERSION}" \
dpcpp-cpp-rt="${IDP_VERSION}" \
mkl-dpcpp="${IDP_VERSION}" \
Expand Down Expand Up @@ -263,8 +263,8 @@ RUN conda run -n pytorch-gpu pip install --no-deps --no-cache-dir --ignore-insta
'ninja>=1.11.1.1' \
'python-dotenv>=1.0.1' \
'tqdm>=4.66.2' \
'cloud-data-connector>=1.0.3' \
'dataset-librarian>=1.0.4' && \
'cloud-data-connector==1.0.3' \
'dataset-librarian==1.0.4' && \
conda run -n pytorch-gpu pip install --no-cache-dir --ignore-installed \
'transformers>=4.40.2' \
'datasets>=2.19.1' \
Expand Down Expand Up @@ -306,7 +306,6 @@ RUN conda create -yn tensorflow-cpu -c "${INTEL_CHANNEL}" -c conda-forge \
tensorflow="${TF_VERSION}" \
impi-devel="${IMPI_VERSION}" \
'matplotlib-base>=3.4.3' \
'dataset_librarian>=1.0.4' \
'ipython>=8.18.1' \
'ipykernel>=6.29.3' \
'kernda>=0.3.0' \
Expand All @@ -330,6 +329,7 @@ RUN conda run -n tensorflow-cpu pip install --no-cache-dir --ignore-installed \
RUN conda run -n tensorflow-cpu pip install --no-deps --no-cache-dir --ignore-installed \
'tensorflow-hub>=0.16.1' \
'tqdm>=4.66.2' \
'dataset-librarian==1.0.4' \
'cloud-data-connector>=1.0.3' && \
conda clean -y --all

Expand All @@ -345,7 +345,6 @@ RUN conda create -yn tensorflow-gpu -c "${INTEL_CHANNEL}" -c conda-forge \
tensorflow="${TF_VERSION}" \
impi-devel="${IMPI_VERSION}" \
'matplotlib-base>=3.4.3' \
'dataset_librarian>=1.0.4' \
'ipython>=8.18.1' \
'ipykernel>=6.29.3' \
'kernda>=0.3.0' \
Expand All @@ -371,7 +370,8 @@ RUN conda run -n tensorflow-gpu pip install --no-cache-dir --ignore-installed \
RUN conda run -n tensorflow-gpu pip install --no-deps --no-cache-dir --ignore-installed \
'tensorflow-hub>=0.16.1' \
'tqdm>=4.66.2' \
'cloud-data-connector>=1.0.3' && \
'dataset-librarian==1.0.4' \
'cloud-data-connector==1.0.3' && \
conda clean -y --all

FROM deep-learning-python as deep-learning-jupyter
Expand Down Expand Up @@ -429,11 +429,11 @@ ENV UCX_TLS=ud,sm,self

USER dev

RUN conda install -n pytorch-cpu -c intel/label/oneapi -c "${INTEL_CHANNEL}" -c conda-forge \
RUN conda install -n pytorch-cpu -c "${INTEL_CHANNEL}" -c conda-forge \
deepspeed="${DEEPSPEED_VERSION}" \
'tensorboardx>=2.6.2.2'

RUN conda install -n pytorch-gpu -c intel/label/oneapi -c "${INTEL_CHANNEL}" -c conda-forge \
RUN conda install -n pytorch-gpu -c "${INTEL_CHANNEL}" -c conda-forge \
deepspeed="${DEEPSPEED_VERSION}" \
'tensorboardx>=2.6.2.2'

Expand Down
2 changes: 1 addition & 1 deletion preset/deep-learning/docker-compose.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ services:
ICD_VER: 23.43.27642.40-803~22.04
IDP_VERSION: ${IDP_VERSION:-2024.1.0}
IMPI_VERSION: ${IMPI_VERSION:-2021.12}
INTEL_CHANNEL: ${INTEL_CHANNEL:-intel}
INTEL_CHANNEL: ${INTEL_CHANNEL:-https://software.repos.intel.com/python/conda/}
IPEX_CPU_VERSION: ${IPEX_CPU_VERSION:-2.2.0=*cpu*}
IPEX_GPU_VERSION: ${IPEX_GPU_VERSION:-2.1.20=*xpu*}
ITEX_VERSION: ${ITEX_VERSION:-2.15}
Expand Down
2 changes: 1 addition & 1 deletion preset/inference-optimization/docker-compose.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ services:
ICD_VER: 23.43.27642.40-803~22.04
IDP_VERSION: ${IDP_VERSION:-2024.1.0}
IMPI_VERSION: ${IMPI_VERSION:-2021.12}
INTEL_CHANNEL: ${INTEL_CHANNEL:-intel}
INTEL_CHANNEL: ${INTEL_CHANNEL:-https://software.repos.intel.com/python/conda/}
IPEX_CPU_VERSION: ${IPEX_CPU_VERSION:-2.2.0=*cpu*}
IPEX_GPU_VERSION: ${IPEX_GPU_VERSION:-2.1.20=*xpu*}
ITEX_VERSION: ${ITEX_VERSION:-2.15}
Expand Down
2 changes: 1 addition & 1 deletion python/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ RUN wget --progress=dot:giga --no-check-certificate "https://github.com/conda-fo
ln -s /opt/conda ~/miniforge3 && \
export PATH="/opt/conda/bin/:${PATH}" && \
conda update -y conda && \
conda config --add channels intel && \
conda config --add channels https://software.repos.intel.com/python/conda/ && \
conda create -yn idp \
"intelpython3_${IDP_VERSION}" \
"python=${PYTHON_VERSION}" && \
Expand Down
3 changes: 2 additions & 1 deletion tensorflow/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -257,7 +257,8 @@ RUN no_proxy="" NO_PROXY="" apt-get update && \
ARG TF_VER="2.15.0"

RUN conda install intel-extension-for-tensorflow=${TF_VER}=*xpu* \
-c intel/label/oneapi -c intel -c conda-forge
-c https://software.repos.intel.com/python/conda \
-c conda-forge

ENV LD_LIBRARY_PATH=/opt/conda/envs/idp/lib:$LD_LIBRARY_PATH

Expand Down
Loading