[ACTION NEEDED] Fix flaky integration tests at distribution level

[gaiksaya]

What is the bug?
It was observed in 2.13.0 and previous other releases that this component manually signed off on the release for failing integration tests. See opensearch-project/opensearch-build#4433 (comment)
The flakiness of the test runs take a lot of time from the release team to collect go/no-go decision and significantly lower the confidence in the release bundles.

How can one reproduce the bug?
Steps to reproduce the behavior:

1. Run integration testing for altering and see the failures.
2. Issues can be reproduced using the steps declared in AUTOCUT issues for failed integration testing

What is the expected behavior?
Tests should be consistently passing.

Do you have any additional context?
Please note that this is a hard blocker for 2.14.0 release as per the discussion here

[ashwin-pc]

Flakey test from 2.13: #6373, opensearch-project/opensearch-dashboards-functional-test#1210

[bbarani]

@ashwin-pc @AMoo-Miki Can you please provide your inputs?

[ashwin-pc]

Both the tests should already be resolved. Will let @kavilla reply once he verifies it

[gaiksaya]

Adding 2.14.0 release manager @rishabh6788

[kavilla]

Checked previously and passing will make check when RC is ran.

[gaiksaya]

Re-tagging the issue with upcoming 2.19 release. Please close if the issue is resolved. Thanks!

[ashwin-pc]

@kavilla can we close this?

[gaiksaya]

I think we are still seeing flaky tests for ci-group 2 and ci-group 6
Adding few release managers that can add more. @rishabh6788 @peterzhuamazon

[ashwin-pc]

Can you link to the specific test failures? We are also tracking CI group failures across OSD core. I want to see if there is an overlap

[gaiksaya]

See the dashboard here for all the tests data. I tried filtering it for 2.19.0 ci-group-6

For 3.0.0-alpha1, @peterzhuamazon is yet to run the tests I believe. Informed it offline. He will update once those results are in.

[peterzhuamazon]

See the dashboard here for all the tests data. I tried filtering it for 2.19.0 ci-group-6

For 3.0.0-alpha1, @peterzhuamazon is yet to run the tests I believe. Informed it offline. He will update once those results are in.

Just an update, 3.0.0 is still pending a fix for opensearch-project/security-analytics-dashboards-plugin#1185 so we are not able to run any OSD tests.

We are working closely with the plugin owner of SA Dash to fix that.

Thanks.

[ashwin-pc]

@gaiksaya based on the dashboard you shared, Ci group 6 has been stable for a few weeks now. Anything im missing?

[gaiksaya]

We did not run 2.19.0 tests after it was released. Last run was on Feb 10, 2025
You can change the version filter and see for 2.19.1.

Here is similar one for current release 3.0.0-alpha1 I believe we are yet to run OSD tests due to SA blocker: opensearch-project/security-analytics-dashboards-plugin#1185

[peterzhuamazon]

3.0.0-alpha1 example:
https://build.ci.opensearch.org/blue/organizations/jenkins/integ-test-opensearch-dashboards/detail/integ-test-opensearch-dashboards/7080/pipeline/157