CI: Ignore failures during UCX testing

[markbrown314]

There two issues with UCX in CI #1148 and #1048 that are under debug. Temporarily disable the test until these tickets are root caused and resolved inorder to reduce noise in CI results.

[markbrown314]

UCX testing in CI mostly fails which leads to noise in the CI results. I propose disabling the tests temporarily until the issue is root caused.

[avincigu]

I am a little bit against this. The UCX tests actually found bugs when I was developing the scan algorithms. We know the signature for the bugs, and though it is very annoying, we can usually just rerun the test so it passes.

[markbrown314]

I am a little bit against this. The UCX tests actually found bugs when I was developing the scan algorithms. We know the signature for the bugs, and though it is very annoying, we can usually just rerun the test so it passes.

Ok, that is fair. The failure occurs intermittently and always occurs in the UCX cma endpoint flush routine during shmem_transport_fini(). This indicates there is some form of memory corruption occurring, but it seems to only be reproducible in the GitHub CI test harness.

How would you feel about this? I could define a configure flag to disable the flush on close in UCX transport (by disabling the UCP_EP_CLOSE_MODE_FLUSH flag from the call to ucp_ep_close_nb()). This disable flag would be enabled during the CI testing until we root cause the issue. This way the UCX testing still occurs but that specific bug is masked.

[avincigu]

I am a little bit against this. The UCX tests actually found bugs when I was developing the scan algorithms. We know the signature for the bugs, and though it is very annoying, we can usually just rerun the test so it passes.

Ok, that is fair. The failure occurs intermittently and always occurs in the UCX cma endpoint flush routine during shmem_transport_fini(). This indicates there is some form of memory corruption occurring, but it seems to only be reproducible in the GitHub CI test harness.

How would you feel about this? I could define a configure flag to disable the flush on close in UCX transport (by disabling the UCP_EP_CLOSE_MODE_FLUSH flag from the call to ucp_ep_close_nb()). This disable flag would be enabled during the CI testing until we root cause the issue. This way the UCX testing still occurs but that specific bug is masked.

If it is easy enough to do, I would very much prefer this solution, however, only if you think it is a simple fix. If it involves a lot of work, forget about it and we can disable the whole thing.

[abrooks98]

In my opinion, disabling an entire section in CI should be the last possible resort. Either adding a config to disable the failing portion or adding the ability to skip specific tests during make check should be explored first

[bcmIntc]

Another option may be to add an exception framework to CI for 'known issues' - when detected re-test. That way we don't bury a bug.

[markbrown314]

Github actions has job setting that will convert errors into warnings: continue-on-error.

This might be a good compromise. All of the tests are run, but erroneous results for UCX's make check are converted into warnings and are annotated in the results output.

[abrooks98]

Thanks for the compromise. I would suggest rewording the PR title, otherwise LGTM

[markbrown314]

Thanks for the compromise. I would suggest rewording the PR title, otherwise LGTM

Good catch!