Skip to content

Commit 2b0a910

Browse files
andistormandistorm
authored
Refactor docker images and workflows to build and deploy them (#22)
* Refactor docker images and workflows to build and deploy them * Remove workflow `deploy-build-kit.yaml`. It is integrated into `deploy-docker-images.yml`. * Refactor deploy-docker-images.yml workflow * Convert matrix into multiple jobs, one for each docker image, because the images depend on each other, and the matrix strategy does not support this. * Add `env-setup` job to set up the environment for the other jobs, because reusable workflows don't support environment variables. * Add input parameter to build deprecated images. Currently, the deprecated images are built by default, if not specified otherwise. * Use output `one_image_tag_short` as input for the dependent images * Refactor build-single-docker-image.yml * Rename input `docker_directory` to `directory`. Instead of setting the directory that contains all images, it is set now to the directory that contains the Dockerfile for the image to build. * Add input parameter `docker_file_name` to specify the Dockerfile to use. * Add input parameter `platforms` to specify the platforms to build the image for. The default is `linux/amd64,linux/arm64` and `linux/arm/v7`. * Add input parameter `depends_on_paths` to specify the paths that the image depends on. If the paths change, the image is rebuilt. * Add input parameter `build_args` to specify the build arguments for the image. * The name of the deployed image is now set by the input parameter `image_name`: `ghcr.io/everest/<image_name>`. The directory name is independent of the image name. * Add output parameter `rebuild` to indicate if the image was rebuilt. * Add output parameter `image_tag_short` to pass the image tag to the dependent images. * Add output parameter `image_tag_long` to pass the full image tag to the dependent images. * Add output parameter `image_tags` to pass all full image tags. * Fix rebuild conditions * Update README.md * Deprecate `ghcr.io/everest/everest-clang-format` image. Its features are now included in `ghcr.io/everest/everest-ci/build-env-base` The old image is deprecated, but still available and deployed. * Merge `ghcr.io/everest/build-kit-alpine` and `ghcr.io/everest/build-kit-debian` into `ghcr.io/everest/everest-ci/build-kit-base`. The old images are deprecated, but still available and deployed. * Add `ghcr.io/everest/everest-ci/run-env-base` image * Add `ghcr.io/everest/everest-ci/build-env-base` image * Add `ghcr.io/everest/everest-ci/dev-env-base` image * Add `run-clang-format.py` script to `build-env-base` image * Update github action `run-glang-format` to use new `build-env-base` image Signed-off-by: Andreas Heinrich <andreas.heinrich@rwth-aachen.de> * Apply suggested changes from review comments Signed-off-by: Andreas Heinrich <andreas.heinrich@rwth-aachen.de> * Apply more suggested changes from review comments Signed-off-by: Andreas Heinrich <andreas.heinrich@rwth-aachen.de> * * Move python3-build requirement from everest-core/build-kit-everest-core to everest-ci/build-env-base * Move documentation related build tools from everest/build-env-everest to everest-ci/build-env-base * Remove documentation related build tools from everest-ci/dev-env-base, since they are included in everest-ci/build-env-base Signed-off-by: Andreas Heinrich <andreas.heinrich@rwth-aachen.de> --------- Signed-off-by: Andreas Heinrich <andreas.heinrich@rwth-aachen.de>
1 parent 5829949 commit 2b0a910

21 files changed

+1079
-74
lines changed

‎.github/workflows/deploy-build-kit.yaml

-44
This file was deleted.

‎.github/workflows/deploy-docker-images.yml

+174-10
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,10 @@ on:
77
description: 'Force rebuild of all images'
88
default: false
99
type: boolean
10+
build_deprecated_images:
11+
description: 'Build deprecated images'
12+
default: false
13+
type: boolean
1014
push:
1115
branches:
1216
- '**'
@@ -15,21 +19,181 @@ on:
1519

1620
env:
1721
REGISTRY: ghcr.io
22+
DOCKER_DIRECTORY: docker/images
23+
PLATFORMS: |
24+
linux/amd64
25+
PATH_TO_DEPLOY_SINGLE_DOCKER_IMAGE_WORKFLOW: .github/workflows/deploy-single-docker-image.yml
26+
PATH_TO_DEPLOY_DOCKER_IMAGES_WORKFLOW: .github/workflows/deploy-docker-images.yml
1827

1928
jobs:
20-
build-and-push-all-images:
21-
name: Build and push all docker images
22-
strategy:
23-
matrix:
24-
image_name: [everest-clang-format]
25-
uses: everest/everest-ci/.github/workflows/deploy-single-docker-image.yml@v1.1.0
29+
env-setup:
30+
# Since env variables can't be passed to reusable workflows, we need to pass them as outputs
31+
name: Evaluate force rebuild and set env variables as outputs
32+
runs-on: ubuntu-22.04
33+
outputs:
34+
force_rebuild: ${{ steps.check.outputs.force_rebuild }}
35+
docker_registry: ${{ env.REGISTRY }}
36+
docker_directory: ${{ env.DOCKER_DIRECTORY }}
37+
platforms: ${{ env.PLATFORMS }}
38+
repository_name: ${{ github.event.repository.name }}
39+
build_deprecated_images: ${{ steps.check.outputs.build_deprecated_images }}
40+
path_to_deploy_single_docker_image_workflow: ${{ env.PATH_TO_DEPLOY_SINGLE_DOCKER_IMAGE_WORKFLOW }}
41+
path_to_deploy_docker_images_workflow: ${{ env.PATH_TO_DEPLOY_DOCKER_IMAGES_WORKFLOW }}
42+
steps:
43+
- id: check
44+
run: |
45+
echo "force_rebuild=${{ inputs.force_rebuild || (github.event_name == 'push' && startsWith(github.event.ref, 'refs/tags/' )) || false }}" >> $GITHUB_OUTPUT
46+
# Build deprecated images if not explicitly disabled
47+
#TODO: set default to false, once backwards compatibility is no longer needed
48+
echo "build_deprecated_images=${{ inputs.build_deprecated_images || true }}" >> $GITHUB_OUTPUT
49+
50+
# One job for each image, since the images build on top of each other a matrix strategy is not possible
51+
run-env-base:
52+
needs:
53+
- env-setup
54+
name: Build and push run-env-base docker image
55+
uses: ./.github/workflows/deploy-single-docker-image.yml
56+
secrets:
57+
SA_GITHUB_PAT: ${{ secrets.SA_GITHUB_PAT }}
58+
SA_GITHUB_USERNAME: ${{ secrets.SA_GITHUB_USERNAME }}
59+
with:
60+
force_rebuild: ${{ needs.env-setup.outputs.force_rebuild == 'true' }}
61+
image_name: ${{ needs.env-setup.outputs.repository_name }}/run-env-base
62+
directory: ${{ needs.env-setup.outputs.docker_directory }}/run-env-base
63+
docker_registry: ${{ needs.env-setup.outputs.docker_registry }}
64+
github_ref_before: ${{ github.event.before }}
65+
github_ref_after: ${{ github.event.after }}
66+
platforms: ${{ needs.env-setup.outputs.platforms }}
67+
depends_on_paths: |
68+
${{ needs.env-setup.outputs.path_to_deploy_single_docker_image_workflow }}
69+
${{ needs.env-setup.outputs.path_to_deploy_docker_images_workflow }}
70+
build-env-base:
71+
needs:
72+
- env-setup
73+
- run-env-base
74+
name: Build and push build-env-base docker image
75+
uses: ./.github/workflows/deploy-single-docker-image.yml
76+
secrets:
77+
SA_GITHUB_PAT: ${{ secrets.SA_GITHUB_PAT }}
78+
SA_GITHUB_USERNAME: ${{ secrets.SA_GITHUB_USERNAME }}
79+
with:
80+
force_rebuild: ${{ needs.env-setup.outputs.force_rebuild == 'true' || needs.run-env-base.outputs.rebuild == 'true' }}
81+
image_name: ${{ needs.env-setup.outputs.repository_name }}/build-env-base
82+
directory: ${{ needs.env-setup.outputs.docker_directory }}/build-env-base
83+
docker_registry: ${{ needs.env-setup.outputs.docker_registry }}
84+
github_ref_before: ${{ github.event.before }}
85+
github_ref_after: ${{ github.event.after }}
86+
platforms: ${{ needs.env-setup.outputs.platforms }}
87+
depends_on_paths: |
88+
${{ needs.env-setup.outputs.path_to_deploy_single_docker_image_workflow }}
89+
${{ needs.env-setup.outputs.path_to_deploy_docker_images_workflow }}
90+
build_args: |
91+
BASE_IMAGE_TAG=${{ needs.run-env-base.outputs.one_image_tag_short }}
92+
93+
dev-env-base:
94+
needs:
95+
- env-setup
96+
- build-env-base
97+
name: Build and push dev-env-base docker image
98+
uses: ./.github/workflows/deploy-single-docker-image.yml
99+
secrets:
100+
SA_GITHUB_PAT: ${{ secrets.SA_GITHUB_PAT }}
101+
SA_GITHUB_USERNAME: ${{ secrets.SA_GITHUB_USERNAME }}
26102
with:
27-
force_rebuild: ${{ inputs.force_rebuild || (github.event_name == 'push' && startsWith(github.event.ref, 'refs/tags/' )) || false }}
28-
image_name: ${{ matrix.image_name }}
29-
docker_directory: docker/images/
30-
docker_registry: ghcr.io
103+
force_rebuild: ${{ needs.env-setup.outputs.force_rebuild == 'true' || needs.build-env-base.outputs.rebuild == 'true' }}
104+
image_name: ${{ needs.env-setup.outputs.repository_name }}/dev-env-base
105+
directory: ${{ needs.env-setup.outputs.docker_directory }}/dev-env-base
106+
docker_registry: ${{ needs.env-setup.outputs.docker_registry }}
31107
github_ref_before: ${{ github.event.before }}
32108
github_ref_after: ${{ github.event.after }}
109+
platforms: ${{ needs.env-setup.outputs.platforms }}
110+
depends_on_paths: |
111+
${{ needs.env-setup.outputs.path_to_deploy_single_docker_image_workflow }}
112+
${{ needs.env-setup.outputs.path_to_deploy_docker_images_workflow }}
113+
build_args: |
114+
BASE_IMAGE_TAG=${{ needs.build-env-base.outputs.one_image_tag_short }}
115+
build-kit-base:
116+
needs:
117+
- env-setup
118+
- build-env-base
119+
name: Build and push build-kit docker image
120+
uses: ./.github/workflows/deploy-single-docker-image.yml
33121
secrets:
34122
SA_GITHUB_PAT: ${{ secrets.SA_GITHUB_PAT }}
35123
SA_GITHUB_USERNAME: ${{ secrets.SA_GITHUB_USERNAME }}
124+
with:
125+
force_rebuild: ${{ needs.env-setup.outputs.force_rebuild == 'true' || needs.build-env-base.outputs.rebuild == 'true' }}
126+
image_name: ${{ needs.env-setup.outputs.repository_name }}/build-kit-base
127+
directory: ${{ needs.env-setup.outputs.docker_directory }}/build-kit-base
128+
docker_registry: ${{ needs.env-setup.outputs.docker_registry }}
129+
github_ref_before: ${{ github.event.before }}
130+
github_ref_after: ${{ github.event.after }}
131+
platforms: ${{ needs.env-setup.outputs.platforms }}
132+
depends_on_paths: |
133+
${{ needs.env-setup.outputs.path_to_deploy_single_docker_image_workflow }}
134+
${{ needs.env-setup.outputs.path_to_deploy_docker_images_workflow }}
135+
build_args: |
136+
BASE_IMAGE_TAG=${{ needs.build-env-base.outputs.one_image_tag_short }}
137+
# Include deprecated images for backwards compatibility
138+
deprecated-everest-clang-format:
139+
if: ${{ needs.env-setup.outputs.build_deprecated_images == 'true' }}
140+
needs:
141+
- env-setup
142+
name: Build and push deprecated everest-clang-format docker image
143+
uses: ./.github/workflows/deploy-single-docker-image.yml
144+
secrets:
145+
SA_GITHUB_PAT: ${{ secrets.SA_GITHUB_PAT }}
146+
SA_GITHUB_USERNAME: ${{ secrets.SA_GITHUB_USERNAME }}
147+
with:
148+
force_rebuild: ${{ needs.env-setup.outputs.force_rebuild == 'true' }}
149+
image_name: everest-clang-format
150+
directory: docker/deprecated-images/everest-clang-format
151+
docker_registry: ${{ needs.env-setup.outputs.docker_registry }}
152+
github_ref_before: ${{ github.event.before }}
153+
github_ref_after: ${{ github.event.after }}
154+
platforms: ${{ needs.env-setup.outputs.platforms }}
155+
depends_on_paths: |
156+
${{ needs.env-setup.outputs.path_to_deploy_single_docker_image_workflow }}
157+
${{ needs.env-setup.outputs.path_to_deploy_docker_images_workflow }}
158+
deprecated-build-kit-alpine:
159+
if: ${{ needs.env-setup.outputs.build_deprecated_images == 'true' }}
160+
needs:
161+
- env-setup
162+
name: Build and push deprecated build-kit-alpine docker image
163+
uses: ./.github/workflows/deploy-single-docker-image.yml
164+
secrets:
165+
SA_GITHUB_PAT: ${{ secrets.SA_GITHUB_PAT }}
166+
SA_GITHUB_USERNAME: ${{ secrets.SA_GITHUB_USERNAME }}
167+
with:
168+
force_rebuild: ${{ needs.env-setup.outputs.force_rebuild == 'true' }}
169+
image_name: build-kit-alpine
170+
directory: docker/deprecated-images/build-kit
171+
docker_file_name: alpine.Dockerfile
172+
docker_registry: ${{ needs.env-setup.outputs.docker_registry }}
173+
github_ref_before: ${{ github.event.before }}
174+
github_ref_after: ${{ github.event.after }}
175+
platforms: ${{ needs.env-setup.outputs.platforms }}
176+
depends_on_paths: |
177+
${{ needs.env-setup.outputs.path_to_deploy_single_docker_image_workflow }}
178+
${{ needs.env-setup.outputs.path_to_deploy_docker_images_workflow }}
179+
deprecated-build-kit-debian:
180+
if: ${{ needs.env-setup.outputs.build_deprecated_images == 'true' }}
181+
needs:
182+
- env-setup
183+
name: Build and push deprecated build-kit-debian docker image
184+
uses: ./.github/workflows/deploy-single-docker-image.yml
185+
secrets:
186+
SA_GITHUB_PAT: ${{ secrets.SA_GITHUB_PAT }}
187+
SA_GITHUB_USERNAME: ${{ secrets.SA_GITHUB_USERNAME }}
188+
with:
189+
force_rebuild: ${{ needs.env-setup.outputs.force_rebuild == 'true' }}
190+
image_name: build-kit-debian
191+
directory: docker/deprecated-images/build-kit
192+
docker_file_name: debian.Dockerfile
193+
docker_registry: ${{ needs.env-setup.outputs.docker_registry }}
194+
github_ref_before: ${{ github.event.before }}
195+
github_ref_after: ${{ github.event.after }}
196+
platforms: ${{ needs.env-setup.outputs.platforms }}
197+
depends_on_paths: |
198+
${{ needs.env-setup.outputs.path_to_deploy_single_docker_image_workflow }}
199+
${{ needs.env-setup.outputs.path_to_deploy_docker_images_workflow }}

0 commit comments

Comments
 (0)