ci(build-and-test-differential): remove sync for the build-and-test-differential workflows

[xmfcx]

Description

Directly related sync job:

• chore: sync files #4372

Proposal

To me, with the latest changes introduced by us, the sync job has grown way too much in complexity to be easily understood by anyone.
I think it is time to cut off the sync job at least for the build-and-test-differential.yaml workflow.

At least until we find a better way to sync this job. Or maybe once we make the containers smaller, we can remove the self hosted option too.

Function of this sync job

It syncs from:

• https://github.com/autowarefoundation/autoware_common/blob/main/.github/workflows/build-and-test-differential.yaml
  to:
• https://github.com/autowarefoundation/autoware.universe/blob/main/.github/workflows/build-and-test-differential.yaml

But while syncing, it makes some brittle changes to the original file as instructed in the sync job file:

• https://github.com/autowarefoundation/autoware.universe/blob/main/.github/sync-files.yaml#L45-L58

ChatGPT analysis of the current sync job

According to ChatGPT analysis, the rename operations can be summarized as follows:

The pre-commands in the second YAML file are using sd (substitute) command, which is a tool used for substituting or replacing strings in files, somewhat similar to sed command in Unix.

1. First Command:

   sd "container: ros:(\w+)" "container: ghcr.io/autowarefoundation/autoware-universe:\$1-latest" {source}

   It substitutes all occurrences of container: ros:<any_word_character> with container: ghcr.io/autowarefoundation/autoware-universe:<any_word_character>-latest.
   Here <any_word_character> is captured by (\w+) and replaced by \$1.
   This change is applied to the {source}, referring to .github/workflows/build-and-test-differential.yaml.

2. Second Command:

   sd -s 'container: ${{ matrix.container }}' 'container: ${{ matrix.container }}${{ matrix.container-suffix }}' {source}

   It substitutes the first occurrence of container: ${{ matrix.container }} with container: ${{ matrix.container }}${{ matrix.container-suffix }} in {source}, adding a potential suffix to the container matrix.

3. Third Command:

   sd -- \
   "       include:" \
   "       container-suffix:
             - \"\"
             - -cuda
           include:" {source}

   This substitutes "include:" with a block that introduces a new matrix variable container-suffix with two values, an empty string, and -cuda, followed by the original include:.
   This likely enables running the workflow with and without CUDA support by appending the suffix to the container name.

4. Fourth Command:

   sd "^    container: ghcr.io/autowarefoundation/autoware-universe:(\w+)-latest\$" \
       "    container: ghcr.io/autowarefoundation/autoware-universe:\$1-latest-cuda" {source}

   It substitutes all occurrences of container: ghcr.io/autowarefoundation/autoware-universe:<any_word_character>-latest at the beginning of a line with container: ghcr.io/autowarefoundation/autoware-universe:<any_word_character>-latest-cuda,
   effectively appending -cuda to the end of the container names.

In Summary:

These pre-commands are modifying the build-and-test-differential.yaml workflow's container names and introducing a new matrix variable,
likely to adjust the environment in which the jobs are run,
possibly to add support for CUDA and to use specific container images from ghcr.io/autowarefoundation/autoware-universe.

History of the sync job

Here is a brief history of the build-and-test-differential sync job in the Universe repository:

1. The initial file was added to the sync job
   • chore: sync files #356
2. Some post-commands are added to rename the containers
   • ci(sync-files): update settings #956
3. More rename operations are added to also have cuda containers too
   • chore: sync files #968
4. build_depends repos rename operations added
   • chore: sync files #983
5. pre-commands are renamed to post-commands and dest changed to src
   • chore: sync files #1182
6. added no-cuda build with more complicated rename operations
   • ci(build-and-test): add no-cuda build #1307
7. galactic replaced with a variable
   • chore: sync files #1514
8. no-cuda test was disabled
   • ci(build-and-test-differential): disable no-cuda test #1669
9. galactic ci was removed
   • ci: remove Galactic CI workflows #2408
10. "no-cuda test was disabled" was reverted

• revert: revert "ci(build-and-test-differential): disable no-cuda test" #2410

And the manual changes made by me and @mitsudome-r :

• chore(workflows): update build-and-test-differential to use self hosted runners #4364
• ci: add label checks before running self-hosted runner #4368

Tests performed

Not applicable.

Effects on system behavior

Not applicable.

Pre-review checklist for the PR author

The PR author must check the checkboxes below when creating the PR.

• I've confirmed the contribution guidelines.
• The PR follows the pull request guidelines.

In-review checklist for the PR reviewers

The PR reviewers must check the checkboxes below before approval.

• The PR follows the pull request guidelines.

Post-review checklist for the PR author

The PR author must check the checkboxes below before merging.

• There are no open discussions or they are tracked via tickets.

After all checkboxes are checked, anyone who has write access can merge the PR.

[mitsudome-r]

I agree that sync file job has become too complex.
I believe it was originally introduced to make sure that we have common settings among different Autoware related repositories so do you propose to maintain them individually manually?

I was just wondering if you want to find out a simpler way of maintaining similar files across different repositories, or you rather want to maintain them manually.

[xmfcx]

@mitsudome-r

I think we can maintain just the Universe one manually for now.

Once we merge this, there are 3 potential futures:

1. Nothing changes
   • We manually maintain this single file.
2. We don't need self hosted runners anymore because the containers are small again
   • We can revert back these changes.
3. We somehow modify the parent file to have a "more easily modifiable" structure with conditional features maybe.
   • This way we can have many instances of this file with different feature sets enabled.
   • I'm not sure of the best way to achieve this just yet.
   • This would require us to modify all the packages that make use of the build-and-test-differential yaml files.

I'd like to go with option 1 for now since it requires the least effort.

Additional note on "self-hosted" naming scheme in many workflow files:

The build-and-test-differential-self-hosted doesn't make sense in the name anymore. All the current/old self-hosted things might need to be renamed to arm maybe.

Until now, we had all the arm things built in self hosted instances. But now we build x86 things in self hosted instances too.

But this will be so many refactorings all over the place, it is not high priority.

[mitsudome-r]

@xmfcx Thanks.
I'm okay with maintaining them manually.
We could change to use GitHub Runner if the cost is too high, but I think it's okay for now.

For the naming convention for "-self-hosted", you are right that we should probably rename them to "-arm", but I agree with you that it is low priority for now.