Run acceptance tests on fork #97
Conversation
This command will trigger an end-to-end workflow with the external contributor's code.
This file contains the same acceptance test jobs with the following differences: - They only run if the `ok-to-test` command triggered the workflow and a sha has been passed. - They checkout from the external contributor's commit. Lastly, this workflow contains an extra job which updates the status in the PR based on the jobs executed. The result of a job is the parent result of all the matrix variants executed as part of it.
| on: | ||
| repository_dispatch: | ||
| types: [ok-to-test-command] | ||
| name: Run acceptance tests [fork] |
There was a problem hiding this comment.
Is it not possible to have a single workflow file that listens for multiple on statements? One for our commits and one for forks?
There was a problem hiding this comment.
TL;DR - It is possible, but I've decided not to merge the two workflows into one file listening for multiple on statements.
I've chosen separating the two for the following reasons:
- Keeping the differences separate. There are two key difference between the jobs in this workflow (for forks) and the ones for our commits:
- Checkout: For our commits we only checkout the latest version of the branch. For forks, we checkout at a specific commit sha provided by us.
- Conditions: For our commits the jobs are executed as usual. For forks, those will be skipped and later have their checks updated by a manual workflow (defined in this file) that is triggered by us.
- Increased complexity. When merging the two files into one, we'd need to add conditions to our steps and jobs, which can make things more complex and prone to edge cases.
- Keeping things clean. This workflow contains an extra step that is responsible to update the checks. It would be a bit weird to see this skipped every single time we make PRs. Also, knowing what will be executed manually versus in general is a cleaner distinction in separate files.
Now I can see why having two separate files can become a hassle from a maintenance standpoint, but I would take a step further and say that this can be simplified once we simplify each acceptance test job, since all 3 are over 80% identical. However, I would consider doing this outside of this PR.
|
Functionally tested this with @edif2008 on another test Fork and it works as expected. |
|
It might also be worth deduplicating the GitHub action script we run in both test and test-fork.yml using this guide: https://joel-azemar.medium.com/github-actions-and-reusable-workflows-544f43e1b450 It would make it simpler to check the actual new code introduced in the PR. |
Currently an external contributor can't have the acceptance tests run on their PR because
pull_requestdoesn't give access to the secrets needed for them.Therefore, in this PR we create a new workflow that is identical to the one for existing acceptance tests, with the following differences:
/ok-to-test sha="<contributor's latest commit sha>"by one of this repo's maintainers.