Skip to content

Commit 608ea1f

Browse files
sramakintelTyler Titsworthpre-commit-ci[bot]dependabot[bot]sharvil10
authored
add apptainer build functionality for python (#211)
Signed-off-by: Srikanth Ramakrishna <srikanth.ramakrishna@intel.com> Signed-off-by: tylertitsworth <tyler.titsworth@intel.com> Signed-off-by: Tyler Titsworth <tyler.titsworth@intel.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Tyler Titsworth <tyler.titsworth@intel.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Sharvil Shah <sharvil.shah@intel.com>
1 parent b7d2fb8 commit 608ea1f

File tree

6 files changed

+266
-0
lines changed

6 files changed

+266
-0
lines changed

‎.github/dependabot.yml

+8
Original file line numberDiff line numberDiff line change
@@ -87,3 +87,11 @@ updates:
8787
package-ecosystem: pip
8888
schedule:
8989
interval: weekly
90+
- directory: /apptainer/python
91+
groups:
92+
apptainer:
93+
patterns:
94+
- "*"
95+
package-ecosystem: pip
96+
schedule:
97+
interval: weekly

‎.github/workflows/apptainer-ci.yaml

+85
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,85 @@
1+
# Copyright (c) 2024 Intel Corporation
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
name: Apptainer CI
16+
on:
17+
pull_request:
18+
paths:
19+
- 'apptainer/**'
20+
21+
concurrency:
22+
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
23+
cancel-in-progress: true
24+
25+
permissions: read-all
26+
27+
jobs:
28+
group-diff:
29+
runs-on: ${{ github.repository_owner == 'intel' && 'intel-ubuntu-latest' || 'ubuntu-latest' }}
30+
outputs:
31+
groups: ${{ steps.group-list.outputs.FOLDERS }}
32+
steps:
33+
- uses: actions/checkout@v4
34+
with:
35+
fetch-depth: 0
36+
- name: Output Modified Group Directories
37+
id: group-list
38+
run: |
39+
# Get diff array filtered by specific filetypes
40+
DIFF=$(git diff --diff-filter=d \
41+
--name-only ${{ github.event.pull_request.base.sha }}...${{ github.event.pull_request.head.sha }} \
42+
-- '*.yaml' '*.yml' '*.def' | \
43+
jq -R '.' | jq -sc '.' \
44+
)
45+
# Search for compose files in each file to determine the container groups
46+
DOCKER_COMPOSE_PATHS=()
47+
for path in $(echo $DIFF | jq -r '.[]'); do
48+
while [[ "$path" != "." ]]; do
49+
DIR_PATH=$(dirname "$path")
50+
if [ -n "$(find "$DIR_PATH" -maxdepth 1 -name 'apptainer.def' -print -quit)" ] && [ "$DIR_PATH" != "." ]; then
51+
DOCKER_COMPOSE_PATHS+=$(echo $"$DIR_PATH" | cut -d'/' -f 2)
52+
path="."
53+
else
54+
path="$DIR_PATH"
55+
fi
56+
done
57+
done
58+
# Convert the array to a JSON array
59+
DOCKER_COMPOSE_PATHS_JSON=$(printf '%s\n' "${DOCKER_COMPOSE_PATHS[@]}" | jq -R '.' | jq -sc 'unique_by(.)')
60+
echo "FOLDERS=$DOCKER_COMPOSE_PATHS_JSON" >> $GITHUB_OUTPUT
61+
62+
setup-build:
63+
needs: [group-diff]
64+
runs-on: ${{ github.repository_owner == 'intel' && 'intel-ubuntu-latest' || 'ubuntu-latest' }}
65+
if: needs.group-diff.outputs.groups != '[""]'
66+
strategy:
67+
matrix:
68+
group: ${{ fromJson(needs.group-diff.outputs.groups) }}
69+
experimental: [true]
70+
fail-fast: false
71+
steps:
72+
- uses: actions/checkout@v4
73+
- uses: eWaterCycle/setup-apptainer@v2.0.0
74+
with:
75+
apptainer-version: 1.3.3
76+
- name: Apptainer login to registry
77+
env:
78+
APPTAINER_DOCKER_USERNAME: ${{ secrets.REGISTRY_USER }}
79+
APPTAINER_DOCKER_PASSWORD: ${{ secrets.REGISTRY_TOKEN }}
80+
run: apptainer registry login -u ${{ secrets.REGISTRY_USER }} -p ${{ secrets.REGISTRY_TOKEN }} docker://${{ secrets.REGISTRY }}
81+
- uses: intel/ai-containers/apptainer@main
82+
with:
83+
registry: oras://${{ secrets.REGISTRY }}
84+
repo: ${{ secrets.REPO }}-apptainer
85+
group_dir: ${{ matrix.group }}

‎apptainer/README.md

+69
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
# Apptainer
2+
3+
Apptainer (formerly Singularity) is an [open-source](https://github.com/apptainer/apptainer) container platform that allows users to build and run containers in a simple, portable and secure way. It is part of the [Linux Foundation](https://apptainer.org/news/community-announcement-20211130/).
4+
5+
## Install Apptainer
6+
7+
You will need a linux system to install and use apptainers. You can install Apptainer either from source or from pre-built packages. Refer to installation instructions provided [here](https://apptainer.org/docs/admin/main/installation.html).
8+
9+
## Project Setup
10+
11+
Define your project's registry and repository each time you use the project:
12+
13+
```bash
14+
export REGISTRY=<registry_name>
15+
export REPO=<repo_name>
16+
17+
apptainer registry login docker://${REGISTRY}
18+
19+
# Verify your access permissions
20+
apptainer pull oras://$REGISTRY/$REPO:latest-sif
21+
```
22+
23+
You can refer to the [link](https://apptainer.org/docs/user/latest/docker_and_oci.html#containers-from-other-registries) for
24+
Apptainer support for different container registries.
25+
26+
## Build and Push Apptainer
27+
28+
To build and push an apptainer container with python ingredient you can do the following:
29+
30+
```bash
31+
cd apptainer/python
32+
apptainer build apptainer.sif apptainer.def
33+
```
34+
35+
To configure the container, use `--build-arg` to override default build arguments provided in the `%arguments` section of the apptainer definition file. For example:
36+
37+
```bash
38+
export BASE_IMAGE_TAG=<specific OS base image tag>
39+
export PYTHON_VERSION=<specific python version>
40+
41+
apptainer build --build-arg BASE_IMAGE_TAG=$BASE_IMAGE_TAG \
42+
--build-arg PYTHON_VERSION=$PYTHON_VERSION \
43+
apptainer-python.sif apptainer.def
44+
apptainer push apptainer-python.sif oras://$REGISTRY/$REPO:python-sif
45+
```
46+
47+
## GitHub Actions CI/CD
48+
49+
The composite action performs build, push and clean up of apptainer images for a given `input_dir` directory.
50+
51+
Inputs for the actions:
52+
53+
```bash
54+
inputs:
55+
group_dir:
56+
description: Directory to build
57+
required: true
58+
type: string
59+
registry:
60+
description: Container Registry URL
61+
required: true
62+
type: string
63+
repo:
64+
description: Container Project Repository
65+
required: true
66+
type: string
67+
```
68+
69+
See an [Example](../.github/workflows/apptainer-ci.yaml#L62) implementation of the action.

‎apptainer/action.yml

+48
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
# Copyright (c) 2024 Intel Corporation
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
---
16+
name: Build Apptainer Group
17+
description: Build Apptainer containers based on defintion files in the apptainer directory
18+
author: srikanth.ramakrishna@intel.com
19+
inputs:
20+
group_dir:
21+
description: Directory to build
22+
required: true
23+
type: string
24+
registry:
25+
description: Container Registry URL
26+
required: true
27+
type: string
28+
repo:
29+
description: Container Project Repository
30+
required: true
31+
type: string
32+
runs:
33+
using: composite
34+
steps:
35+
- name: Build Apptainer
36+
shell: bash
37+
run: |
38+
apptainer build "${{ inputs.group_dir }}.sif" apptainer.def
39+
working-directory: apptainer/${{ inputs.group_dir }}
40+
- name: Push Apptainer
41+
shell: bash
42+
run: |
43+
apptainer push ${{ inputs.group_dir }}.sif "${{ inputs.registry }}/${{ inputs.repo }}:${{ github.run_number }}-${{ inputs.group_dir }}-sif"
44+
working-directory: apptainer/${{ inputs.group_dir }}
45+
- name: Clean cache and Remove Images
46+
shell: bash
47+
run: apptainer cache clean --force && rm -rf *.sif
48+
working-directory: apptainer/${{ inputs.group_dir }}

‎apptainer/python/apptainer.def

+50
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
# Copyright (c) 2024 Intel Corporation
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
Bootstrap: docker
16+
From: {{BASE_IMAGE_NAME}}:{{ BASE_IMAGE_TAG}}
17+
18+
%arguments
19+
BASE_IMAGE_NAME=ubuntu
20+
BASE_IMAGE_TAG=22.04
21+
PYTHON_VERSION=3.10
22+
23+
%environment
24+
export DEBIAN_FRONTEND=noninteractive
25+
26+
%files
27+
requirements.txt .
28+
29+
%post
30+
apt-get update -y && \
31+
apt-get install -y --no-install-recommends --fix-missing \
32+
ca-certificates \
33+
wget
34+
35+
apt-get install -y --no-install-recommends --fix-missing \
36+
libpython{{ PYTHON_VERSION }} \
37+
python{{ PYTHON_VERSION }} \
38+
python3-pip \
39+
python{{ PYTHON_VERSION }}-distutils && \
40+
apt-get clean && \
41+
rm -rf /var/lib/apt/lists/*
42+
43+
rm -rf /usr/lib/python{{ PYTHON_VERSION }}/EXTERNALLY-MANAGED
44+
45+
ln -sf "$(which python{{ PYTHON_VERSION }})" /usr/local/bin/python && \
46+
ln -sf "$(which python{{ PYTHON_VERSION }})" /usr/local/bin/python3 && \
47+
ln -sf "$(which python{{ PYTHON_VERSION }})" /usr/bin/python
48+
49+
python -m pip --no-cache-dir install --upgrade pip
50+
python -m pip --no-cache-dir install -r requirements.txt

‎apptainer/python/requirements.txt

+6
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
numpy==1.26.4
2+
setuptools==69.5.1
3+
psutil==6.0.0
4+
mkl==2024.2.0
5+
mkl-include==2024.2.0
6+
intel-openmp==2024.2.0

0 commit comments

Comments
 (0)