Support kernel overrides in ex rebuild container flow

[cgwalters]

We want to be able to override the kernel and have that Just Work when rebasing on the client-side.

This is a follow-up to #3489.

[jlebon]

I think we should probably split this issue into three:

• support rpm-ostree override remove and override-remove dropins
• support rpm-ostree override replace and override-replace dropins for !kernel
• support rpm-ostree override replace and override-replace dropins for kernel, which is more complex

One thing though to clarify is the concept of "override" in the CoreOS layering path. If we have a pipeline consisting of multiple sequential layering black boxes, each of which modifying the package set, do later layering steps consider newly added packages as "base" or are we always comparing against the base OSTree commit?

My intuition tells me we want the former semantic because e.g. from a context where part of that pipeline is vendor-controlled and part is user-controlled, we want vendor "extensions" to be considered base packages so that users think carefully about overriding them vs just having to uninstall. It also makes implementing all this easier I think.

[jlebon]

OK, I've split this in two:

• Support non-kernel overrides in ex rebuild container flow #3489; I've kept the original description of this issue there
• Support kernel overrides in ex rebuild container flow #3364 (this issue); this one will track kernel overrides specifically

[cgwalters]

OK it'd be great to pick this up for Fedora 37/OCP 4.12 timeframe-ish.

Basically here's the thing - so far in the "rpm-ostree-in-container" flow we are 95% deferring to libdnf. But to implement this correctly, we must intercept /bin/kernel-install. Today, rpm-ostree compose tree handles this by running each package script in its own container (via bubblewrap) and we simply know not to execute the scripts for kernel:

rpm-ostree/rust/src/scripts.rs

Line 20 in 0c738fb

// We take over depmod/dracut etc. It's `kernel` in C7 and kernel-core in F25+

But we can't rely on container-in-container stuff here. We can intercept %post scripts but that's a whole big task that involves not using the libdnf machinery for package installs. I think that'd be good long term - it would help unblock various things, including handling translation of /var/lib/foo to tmpfiles.d etc.

But short term I'd say we add kernel-install to our cliwrap.rs - we already support dracut there. Basically, what our kernel-install needs to do is just regenerate the initramfs but drop it in /usr/lib/modules/$kver/initramfs.img and not /boot. We should absolutely not run os-prober or really any grub stuff at all.

[jmarrero]

Will be working on this on the following PR: #3689

[jmarrero]

This is now done via #3689