Work around Docker overlayfs inconsistencies by using rsync

With the new approach of building the images (where the entire final
rootfs is copied into the second stage), if the system building the
containers is using the overlay2 storage driver (which is the default)
and is able to use native diffs (which could be true if
CONFIG_OVERLAY_FS_REDIRECT_DIR isn't enabled in the kernel), then the
final result of the image will be different than if naive diffs (where
Docker compares the metadata of each file and, if needed, the contents
to find out if something has changed) were used. Specifically, with
native diffs, each container would be much larger, since technically
speaking, the whole rootfs is being written to, even if the content ends
up the same. This appears to be a known issue (in some form), and
workarounds are being thought of in moby/moby#35280.

As a workaround, install rsync into the base container, copy the
entirely of that into an empty base image, and use rsync to copy only
the changed files into the layer in one shot. This does mean that rsync
will remain installed in the final built containers, but hopefully this
is fine.

Signe-off-by: Saikrishna Arcot <sarcot@microsoft.com>

Author: saiarcot895

dockers/docker-base-bookworm/Dockerfile.j2

@@ -35,6 +35,8 @@ RUN apt update &&            \
         python-is-python3    \
         vim-tiny             \
         rsyslog              \
+# Install rsync for copying over only changes between layers
+        rsync                \
 # Install redis-tools
         redis-tools          \
 # common dependencies
@@ -94,6 +96,6 @@ RUN ln /usr/bin/vim.tiny /usr/bin/vim
 
 COPY ["etc/supervisor/supervisord.conf", "/etc/supervisor/"]
 
-FROM $BASE
+FROM scratch
 
 COPY --from=base / /

dockers/docker-base-bullseye/Dockerfile.j2

@@ -98,4 +98,4 @@ COPY ["etc/supervisor/supervisord.conf", "/etc/supervisor/"]
 
 FROM $BASE
 
-COPY --from=base / /
+RUN --mount=type=bind,from=base,target=/changes-to-image rsync -axAX --no-D --exclude=sys --exclude=resolv.conf /changes-to-image/ /