Add ramalama image built on Fedora using Fedora's rocm packages #596

maxamillion · 2025-01-16T18:18:28Z

Add ramalama image built on Fedora using Fedora's rocm packages, this enables more models of embedded GPUs in the Ryzen APUs for more series of gfx than is available in the official ROCm packages from AMD that are used in the UBI based default images.

rhatdan · 2025-01-16T18:24:27Z

Do we actually need a separate Containerfile or could we just do this with a podman build --from fedora ... or I guess to make it more container engine agnostic use BUILDARGS.

ericcurtin · 2025-01-16T18:24:41Z

container-images/ramalama-fedora/Containerfile

@@ -0,0 +1,8 @@
+FROM registry.fedoraproject.org/fedora:latest
+
+COPY ../scripts /scripts


I know it's not obvious apologies, but "ramalama" means use a generic driver, aka vulkan/kompute. I think what we could do here is:

/scripts/build_llama_and_whisper.sh "rocm" "fedora"

with the first parameter being the GPU/AI driver (rocm in this case) and the second parameter could be fedora, ubi, etc.

maxamillion · 2025-01-16T19:18:02Z

Do we actually need a separate Containerfile or could we just do this with a podman build --from fedora ... or I guess to make it more container engine agnostic use BUILDARGS.

That's probably a better option, I'll go that route.

ericcurtin · 2025-01-16T19:24:05Z

container-images/scripts/build_llama_and_whisper.sh

@@ -36,7 +36,11 @@ dnf_install() {
    dnf install -y asahi-repos
    dnf install -y mesa-vulkan-drivers "${vulkan_rpms[@]}" "${rpm_list[@]}"
  elif [ "$containerfile" = "rocm" ]; then
-    dnf install -y rocm-dev hipblas-devel rocblas-devel
+    if [ "${ID}" = "fedora" ]; then


Much nicer :)

ericcurtin · 2025-01-16T19:25:14Z

container-images/rocm-fedora/Containerfile

@@ -1,8 +1,10 @@
 FROM registry.fedoraproject.org/fedora:latest

+RUN dnf update -y --refresh && dnf clean all


This step doesn't seem unique to this Containerfile or anything, no big deal though

ericcurtin

Merging on green build

maxamillion · 2025-01-16T19:36:35Z

@ericcurtin would you prefer I converge the two rocm Containerfiles and use --build-args as @rhatdan suggested or would that be a separate PR?

ericcurtin · 2025-01-16T20:01:09Z

@maxamillion I don't mind, if it works, you'll probably have to edit container_build.sh for that technique

ericcurtin · 2025-01-16T21:12:53Z

The build failed because the container image is too large, we probably want to remove some older GPUs from it, for example on the UBI container images we rm'd *gfx9* files to trim the image, these are GPUs prior to ~2018

ericcurtin · 2025-01-16T21:43:52Z

@debarshiray and @owtaylor I remember you guys wanted some Fedora based RamaLama images. Well... It's coming it seems 😄 Thanks to @maxamillion

ericcurtin · 2025-01-17T16:13:35Z

container-images/scripts/build_llama_and_whisper.sh

@@ -115,7 +124,8 @@ main() {
  dnf clean all
  rm -rf /var/cache/*dnf* /opt/rocm-*/lib/llvm \
    /opt/rocm-*/lib/rocblas/library/*gfx9* \
-    /opt/rocm-*/lib/hipblaslt/library/*gfx9* llama.cpp whisper.cpp
+    /opt/rocm-*/lib/hipblaslt/library/*gfx9* llama.cpp whisper.cpp \
+    /usr/lib64/rocm/gfx9* 


This might make it small enough :)

It's actually a restriction on the storage space available on our github runner, but it's a welcome restriction, we don't want 20GB container images

One could always contribute a rocm-legacy-fedora type image for gfx9 if they had the motivation to do so.

We should actually simplify the above line to:

/opt/rocm-*/lib/*/library/*gfx9*

not necessary in this PR or anything.

we'll find out if it's small enough soon! 😄

@ericcurtin it appears that the build is still angry about it 🤔

Yeah I guess you just gotta "podman run --rm -it bash" into it and look around and see what's big, gfx9 was enough to remove for the UBI versions. Trim, test to make sure you didn't break the ROCm support, then add the "rm -fr" here.

@ericcurtin question - what's the size goal I'm trying to hit?

I can't remember exactly, these ones pushed here are built in this CI and are within the limit (https://quay.io/repository/ramalama/rocm?tab=tags). The ROCm images are the only images that suffer from this, I've never had to trim a CUDA one for example.

It's arguably worth logging an issue in ROCm upstream, it's unlikely all those *gfx* files we trim are completely unique, I'm sure there's loads of duplicate data in those files that could be shared using some common files or some other form of de-duplication.

It leaves a bad first impression for ramalama if users have to pull huge container images just to get going, some people won't wait around for hours for a large image to be pulled.

We could also do container images like rocm-gfx9-fedora, rocm-gfx10-fedora, etc. We may be able to detect what generation we are pre-pull

ericcurtin · 2025-01-17T20:03:37Z

UBI has nothing lower than gfx9 @maxamillion I see you found gfx8 . If there's gfx7 or less, likely worth removing also.

maxamillion · 2025-01-17T20:18:45Z

@ericcurtin nothing older than gfx8, but it's still a lot of space (granted, it was 14G before I pulled both gfx9* and gfx8* out)

[root@a854647ecf79 /]# ls /usr/lib64/rocm
gfx10  gfx11  gfx1100  gfx1103
[root@a854647ecf79 /]# du -sh /usr/lib64/rocm/
5.3G    /usr/lib64/rocm/

ericcurtin · 2025-01-17T20:25:32Z

Looks like you are almost there, if only we could rank the GPU models by units sold or something and trim based on least units sold.

ericcurtin · 2025-01-17T20:27:33Z

Could also be interesting to see if there is further compression we can do other than whatever podman build does by default. @rhatdan any compression techniques, etc. we could do off the top of your head?

ericcurtin · 2025-01-17T20:29:25Z

gfx10 seems like the first models were released in 2019 FWIW

ericcurtin · 2025-01-17T20:31:27Z

Maybe we will be lucky and this will be small enough to go green

ericcurtin · 2025-01-17T20:34:54Z

Seperate images for gfx8 gfx9 gfx10 gfx11 would be a welcome addition to be honest, it would improve UX, faster pulls. We may be able to detect which image we need to pull on a given machine by querying /proc /sys /dev etc. Pre-pull. Also any image with < 1GB VRAM has incredibly low value in enabling, often CPU inferencing is better when VRAM is this low (maybe even < 2GB, 4GB, 8GB VRAM). An 8 GB VRAM GPU is considered small these days.

maxamillion · 2025-01-18T01:34:05Z

well bummer ... even splitting them all out the builds are too large 🤔

ericcurtin · 2025-01-19T19:17:54Z

We could just use the exact same list of GPUs as the UBI9 images and enable additional ones on demand as people request.

maxamillion · 2025-01-22T16:27:38Z

@ericcurtin I don't see a way to trim this down further, I'm open to suggestion though.

This is some sample output from the rocm-fedora-gfx11 image.

[root@167cf24d485c /]# du -sh /usr/* | grep G
8.6G    /usr/lib64

[root@167cf24d485c /]# du -sh /usr/lib64/* | grep G
1.1G    /usr/lib64/librocblas.so.4.2
1.9G    /usr/lib64/librocsparse.so.1.0
3.0G    /usr/lib64/rocm

[root@167cf24d485c /]# du -sh /usr/lib64/rocm/* | grep G
2.3G    /usr/lib64/rocm/gfx11

[root@167cf24d485c /]# du -sh /usr/lib64/rocm/gfx11/* | grep G
2.3G    /usr/lib64/rocm/gfx11/lib

[root@167cf24d485c /]# du -sh /usr/lib64/rocm/gfx11/lib/*
32K     /usr/lib64/rocm/gfx11/lib/cmake
4.0K    /usr/lib64/rocm/gfx11/lib/libhipblas.so
4.0K    /usr/lib64/rocm/gfx11/lib/libhipblas.so.2
996K    /usr/lib64/rocm/gfx11/lib/libhipblas.so.2.2
4.0K    /usr/lib64/rocm/gfx11/lib/librocblas.so
4.0K    /usr/lib64/rocm/gfx11/lib/librocblas.so.4
486M    /usr/lib64/rocm/gfx11/lib/librocblas.so.4.2
4.0K    /usr/lib64/rocm/gfx11/lib/librocsolver.so.0
359M    /usr/lib64/rocm/gfx11/lib/librocsolver.so.0.2
4.0K    /usr/lib64/rocm/gfx11/lib/librocsparse.so.1
830M    /usr/lib64/rocm/gfx11/lib/librocsparse.so.1.0
589M    /usr/lib64/rocm/gfx11/lib/rocblas

Based on the output, there's not really anything in there we can rip out based on specific GPU models. It's all just bundled into giant shared object files 😕

maxamillion · 2025-01-22T16:30:33Z

@ericcurtin in a shocking turn of events, if I set it to rawhide as a base image the rocm-fedora-gf11 image is about 4gb smaller than if I build with Fedora 41 🤷 .... any reservations with switching everything to rawhide for now?

maxamillion · 2025-01-22T16:31:58Z

@ericcurtin in a shocking turn of events, if I set it to rawhide as a base image the rocm-fedora-gf11 image is about 4gb smaller than if I build with Fedora 41 🤷 .... any reservations with switching everything to rawhide for now?

lol nvm .. it's smaller because cmake failed 🤦

rhatdan · 2025-01-22T16:32:06Z

I am fine with that, do you know why it is smaller?

maxamillion · 2025-01-22T16:33:46Z

I am fine with that, do you know why it is smaller?

because cmake failed 🤦

ericcurtin · 2025-01-22T17:52:10Z

No issue with rawhide, eventually we can call it Fedora 42

ericcurtin · 2025-01-22T17:57:01Z

This is an isolated image, so that helps

ericcurtin · 2025-02-04T21:36:13Z

$ grep gfx /sys/class/kfd/kfd/topology/nodes/*/properties
/sys/class/kfd/kfd/topology/nodes/0/properties:gfx_target_version 0
/sys/class/kfd/kfd/topology/nodes/1/properties:gfx_target_version 110002
/sys/class/kfd/kfd/topology/nodes/2/properties:gfx_target_version 90012

there's a little decimal to hex conversion to get "02" and "0c"

Yes, it's:

major_ver = int((device_id / 10000) % 100)
minor_ver = int((device_id / 100) % 100)
stepping_ver = int(device_id % 100)

... and then:

"gfx" + format(major_ver, 'd') + format(minor_ver, 'x') + format(stepping_ver, 'x')

Did you figure out how to filter out your smaller GPU?

Yes it's code elsewhere in RamaLama:

    # ROCm/AMD CASE
    i = 0
    gpu_num = 0
    gpu_bytes = 0
    for fp in sorted(glob.glob('/sys/bus/pci/devices/*/mem_info_vram_total')):
        with open(fp, 'r') as file:
            content = int(file.read())
            if content > 1073741824 and content > gpu_bytes:
                gpu_bytes = content
                gpu_num = i

        i += 1

basically ignore every GPU with less than 1GB VRAM and choose the one with the most VRAM.

ericcurtin · 2025-02-14T21:21:06Z

container-images/scripts/build_llama_and_whisper.sh

-    dnf install -y "$url"
-    crb enable # this is in epel-release, can only install epel-release via url
-    dnf --enablerepo=ubi-9-appstream-rpms install -y "${rpm_list[@]}"
+    if [[ "${ID}" == "rhel" || "${ID}" == "redhat" || "${ID}" == "centos" ]]; then


I think we'd catch all these with:

PLATFORM_ID

Or even ID is not fedora

ericcurtin · 2025-02-14T21:22:12Z

container-images/scripts/build_llama_and_whisper.sh

@@ -1,4 +1,5 @@
 #!/bin/bash
+source /etc/os-release


Normally like to keep everything in main() function

ericcurtin · 2025-02-14T21:23:02Z

I'd merge this only it still has "WIP:" in it. I didn't check the CI logs yet to see if all these were actually built

ericcurtin · 2025-02-14T21:23:25Z

But LGTM in general

ericcurtin · 2025-02-14T21:25:27Z

My suspicions were right, the builds for the container images were turned off, we should turn them on again.

rhatdan · 2025-02-15T10:50:56Z

container-images/rocm-fedora-gfx10/Containerfile

@@ -0,0 +1,11 @@
+FROM registry.fedoraproject.org/fedora:latest


Rather then have 4 Containerfiles, that we need to manage can we just consolidate this into one ROCM containerfile and then manage this with container_build.sh and build_llama_whisper.sh

These containerfiles have very little difference that I can see then which files are being removed for size.

If users say make build IMAGE=rocm-fedora-gfx10 Then this should be enough to change the rm -fr ... line and potentially the FROM line.

I see what you mean @rhatdan I'd just add another argument to the build_llama_and_whisper.sh script if it was me, would remove these lines:

RUN dnf update -y --refresh && dnf clean all ... ENV WHISPER_CPP_SHA=${WHISPER_CPP_SHA} ENV LLAMA_CPP_SHA=${LLAMA_CPP_SHA}

ericcurtin · 2025-02-16T15:20:55Z

PR to build all container image builds in CI:

#831

ericcurtin · 2025-02-16T15:24:44Z

container-images/rocm-fedora-gfx8/Containerfile

@@ -0,0 +1,11 @@
+FROM registry.fedoraproject.org/fedora:latest


Would prefer to tag :latest to a fixed fedora version and let renovate update it.

Signed-off-by: Adam Miller <admiller@redhat.com>

maxamillion · 2025-02-26T01:16:53Z

@ericcurtin @rhatdan sorry for the delay, I finally had time to refactor this based on feedback and it should be ready for review now

rhatdan · 2025-02-26T17:39:51Z

@maxamillion Are you thinking we will have 4 or more ROCM images?

ericcurtin · 2025-02-26T18:07:51Z

container-images/scripts/build_llama_and_whisper.sh

+      ;;
+    gfx11)
+      rm -fr /usr/lib64/rocm/gfx8* /usr/lib64/rocm/gfx9* /usr/lib64/rocm/gfx10* && \
+      ln -s /usr/lib64/rocm/gfx1103/lib/rocblas/library/TensileLibrary_lazy_gfx1103.dat \


Interesting you figured out we need to do this, worth alerting Tom Rix to this, it might be better to fix this in the rpm package. Worth checking if this is needed for UBI9 also

@ericcurtin that isn't needed in the Rawhide and Fedora 42 nightlies of ROCm but support for gfx1103 is considered experimental in Fedora 41 so I'm not sure where the fix is appropriate.

Yes. gfx1103 was experimental in F41.
gfx1035,gfx1103,gfx1150,gfx1151,gfx1152,gfx1200,gfx1201 were added in F42.

maxamillion · 2025-02-26T19:58:41Z

@maxamillion Are you thinking we will have 4 or more ROCM images?

Unfortunately I'm not sure. I wanted it to be all-in-one but the made the container image massive and it was suggested that we break it out so I did. I don't know how many Fedora ROCm images we'll need over time, but I suspect at least one per gfx major release series unless we can find an alternative method of making the ROCm stack smaller in the Fedora builds.

rhatdan · 2025-02-27T15:57:28Z

Ok we can start by building multiple images, but we need to get a naming standard and make the GPU check pull the right image. Also if we can make sure we have as big of a shared image as possible,

ericcurtin · 2025-02-27T22:17:32Z

I dunno by how much or which one, but it appears one of the images is still too big.

maxamillion mentioned this pull request Jan 16, 2025

Compilation errors trying to build a ramalama fedora image #594

Closed

ericcurtin reviewed Jan 16, 2025

View reviewed changes

ericcurtin approved these changes Jan 16, 2025

View reviewed changes

ericcurtin reviewed Jan 17, 2025

View reviewed changes

maxamillion changed the title ~~Add ramalama image built on Fedora using Fedora's rocm packages~~ WIP: Add ramalama image built on Fedora using Fedora's rocm packages Jan 21, 2025

maxamillion force-pushed the fedora-rocm branch from 42a89f7 to b28db8c Compare January 21, 2025 22:58

maxamillion force-pushed the fedora-rocm branch from 42f52ad to 5c16714 Compare February 4, 2025 22:41

maxamillion requested review from bmahabirbu, swarajpande5, jhjaggars, cgruver, slp and engelmi as code owners February 4, 2025 22:41

maxamillion force-pushed the fedora-rocm branch from 5f11a70 to 7a20b3d Compare February 14, 2025 20:46

ericcurtin reviewed Feb 14, 2025

View reviewed changes

rhatdan reviewed Feb 15, 2025

View reviewed changes

ericcurtin reviewed Feb 16, 2025

View reviewed changes

maxamillion added 4 commits February 25, 2025 18:41

Fedora rocm images

549d2ea

Signed-off-by: Adam Miller <admiller@redhat.com>

update to handle vulkan/blas packages for fedora

61a3f0a

Signed-off-by: Adam Miller <admiller@redhat.com>

move source to main() and clean up conditional

0b02681

Signed-off-by: Adam Miller <admiller@redhat.com>

collapse to a single containerfile, refactor build scripts to accomodate

7e0b5d5

Signed-off-by: Adam Miller <admiller@redhat.com>

maxamillion force-pushed the fedora-rocm branch from 7e672c6 to 7e0b5d5 Compare February 26, 2025 00:48

suppress shellcheck issue with source=/etc/os-release

a21e1e5

Signed-off-by: Adam Miller <admiller@redhat.com>

maxamillion changed the title ~~WIP: Add ramalama image built on Fedora using Fedora's rocm packages~~ Add ramalama image built on Fedora using Fedora's rocm packages Feb 26, 2025

ericcurtin reviewed Feb 26, 2025

View reviewed changes

		@@ -0,0 +1,8 @@
		FROM registry.fedoraproject.org/fedora:latest

		COPY ../scripts /scripts

		@@ -1,8 +1,10 @@
		FROM registry.fedoraproject.org/fedora:latest

		RUN dnf update -y --refresh && dnf clean all

		@@ -0,0 +1,11 @@
		FROM registry.fedoraproject.org/fedora:latest

Add ramalama image built on Fedora using Fedora's rocm packages #596

Are you sure you want to change the base?

Add ramalama image built on Fedora using Fedora's rocm packages #596

Conversation

maxamillion commented Jan 16, 2025

rhatdan commented Jan 16, 2025

ericcurtin Jan 16, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

maxamillion commented Jan 16, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ericcurtin left a comment

Choose a reason for hiding this comment

maxamillion commented Jan 16, 2025

ericcurtin commented Jan 16, 2025

ericcurtin commented Jan 16, 2025 • edited Loading

ericcurtin commented Jan 16, 2025 • edited Loading

ericcurtin Jan 17, 2025 • edited Loading

Choose a reason for hiding this comment

ericcurtin Jan 17, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ericcurtin Jan 17, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ericcurtin Jan 17, 2025 • edited Loading

Choose a reason for hiding this comment

ericcurtin Jan 17, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ericcurtin commented Jan 17, 2025 • edited Loading

maxamillion commented Jan 17, 2025

ericcurtin commented Jan 17, 2025

ericcurtin commented Jan 17, 2025 • edited Loading

ericcurtin commented Jan 17, 2025

ericcurtin commented Jan 17, 2025

ericcurtin commented Jan 17, 2025 • edited Loading

maxamillion commented Jan 18, 2025

ericcurtin commented Jan 19, 2025

maxamillion commented Jan 22, 2025

maxamillion commented Jan 22, 2025

maxamillion commented Jan 22, 2025

rhatdan commented Jan 22, 2025

maxamillion commented Jan 22, 2025

ericcurtin commented Jan 22, 2025

ericcurtin commented Jan 22, 2025

ericcurtin commented Feb 4, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ericcurtin commented Feb 14, 2025 • edited Loading

ericcurtin commented Feb 14, 2025

ericcurtin commented Feb 14, 2025

Choose a reason for hiding this comment

ericcurtin Feb 16, 2025 • edited Loading

Choose a reason for hiding this comment

ericcurtin commented Feb 16, 2025

Choose a reason for hiding this comment

maxamillion commented Feb 26, 2025

rhatdan commented Feb 26, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

maxamillion commented Feb 26, 2025

rhatdan commented Feb 27, 2025

ericcurtin commented Feb 27, 2025

ericcurtin Jan 16, 2025 •

edited

Loading

ericcurtin commented Jan 16, 2025 •

edited

Loading

ericcurtin commented Jan 16, 2025 •

edited

Loading

ericcurtin Jan 17, 2025 •

edited

Loading

ericcurtin Jan 17, 2025 •

edited

Loading

ericcurtin Jan 17, 2025 •

edited

Loading

ericcurtin Jan 17, 2025 •

edited

Loading

ericcurtin Jan 17, 2025 •

edited

Loading

ericcurtin commented Jan 17, 2025 •

edited

Loading

ericcurtin commented Jan 17, 2025 •

edited

Loading

ericcurtin commented Jan 17, 2025 •

edited

Loading

ericcurtin commented Feb 4, 2025 •

edited

Loading

ericcurtin commented Feb 14, 2025 •

edited

Loading

ericcurtin Feb 16, 2025 •

edited

Loading