Check, take action or notify and auto-update Entrance criteria for releases

[gaiksaya]

Is your feature request related to a problem? Please describe

Coming from #5171

Just focusing on entrance criteria for the scope of this issue

Currently, the release manager must manually verify the entrance criteria before or during the release window. This process is prone to human error and allows for inconsistent execution of releases, as there is no standardized approach. To streamline the process and eliminate these challenges, it would be ideal for these criteria to be automatically checked, updated, and made actionable.

Describe the solution you'd like

The current entrance criterias as as below:

* Each component release issue has an assigned owner

Check:
Check if all the release component issues have release owners assigned using data in the metrics cluster.

Action:
If not, below are the AI:

• 1 week before the release window opens: Notify by tagging all the maintainers of the repo in the GH comment on the release issue.
• 3 days before the release window opens: If still no release owner is assigned, we will be randomly assigning one from the list of maintainer.

* Documentation draft PRs are up and in tech review for all component changes.

Check:
Check if the PRs are up for all raised issues. Sample query:

is:issue is:open label:v<release_version> -linked:pr 

Action:

• If not, a GH issue comment will be added in the raised issues notifying the requester to either create the PR or revert the feature from the release due to lacking documentation.

* Sanity testing is done for all components.

Check:
No direct way of checking it. We reply on component teams to do the needful. From distribution side, we can run smoke tests and validation for sanity.

* Code coverage has not decreased (all new code has tests).

Check:
We are indexing the code coverage data in the metrics cluster. For given release (eg: 2.19) we can check the difference against the last minor (2.18 in this case).

Action:
Notify the teams by commenting on the GH issue as well as main release issue asking to add tests for missing ones as the code coverage as decreased.
Heads-up: Looks like code coverage reporting is a problem for many components. We can use this opportunity to fix that as well.

* Release notes are ready and available for all components.

Check:
Check using metrics data.

Action:
Tag and notify by commenting on component Release issue asking to fulfill the missing criteria.

* Roadmap is up-to-date (information is available to create release highlights).

We rely on component owners and maintainers to update the roadmap. Wondering if making sure the issues are closed as part of the check makes sense @Pallavi-AWS @getsaurabh02

* Release ticket is cut, and there's a forum post announcing the start of the window.

Check:
Check using metrics data and compare against the release input manifest to match the number of components and issues.

Action:
If some are missing, build/infra repo will create the component release issue accordingly.

* Any necessary security reviews are complete.

We rely on component owners and maintainers to follow through.

[gaiksaya]

Tagging @opensearch-project/engineering-effectiveness to get some inputs or feedback if they have any.
Thanks!

[peterzhuamazon]

As of current, roadmap can be auto-updated with a roadmap: or meta / rfc label, tho we do want to make some adjustment so that everyone who open a meta issue get automatically added to the roadmap, and more.

Each meta should also have corresponding label for version so that they can automatically categorized on roadmap.

Thanks.

[gaiksaya]

See the above linked PRs for tasks completed. There is some ambiguity around the code as the data to take actions is mainly retrieved from the OpenSearch metrics cluster and we need to form queries, write tests for those queries as well as libraries for robustness.

The data in metrics cluster needs to be up to date in order to take action accordingly. With 3.0.0 going on we are facing one of the edge case with the qualifier. See opensearch-project/opensearch-metrics#121

[gaiksaya]

Had a sync up with @prudhvigodithi regarding release notes. We have two frameworks today that can monitor release notes.

1. Python framework : https://github.com/opensearch-project/opensearch-build/tree/main/src/release_notes_workflow
2. Metrics Data: https://github.com/opensearch-project/opensearch-metrics/blob/main/src/main/java/org/opensearchmetrics/metrics/release/ReleaseNotesChecker.java

The difference is python framework checks release notes at the mentioned ref from the input manifest. However, the metrics data tracks the release branch (no matter what manifest says).

We can have a library based on metrics data to check the notes automatically and still use python framework to catch the diff.
Integrating the python framework with groovy codebase and metrics data to make the results actionable.

P.S: There is a bug regarding qualifier part in metrics release notes. Created PR to fix it: opensearch-project/opensearch-metrics#126
Thanks!