Add per-tenant provisioning throttling

[dbwiddis]

Description

Adds settings to control per-tenant limits on simultaneous provision, reprovision, and deprovision.

Check List

• New functionality includes testing.
• New functionality has been documented.
• [ ] API changes companion pull request created.
• Commits are signed per the DCO using --signoff.
• [ ] Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov]

Codecov Report

Attention: Patch coverage is 69.01408% with 22 lines in your changes missing coverage. Please review.

Project coverage is 76.32%. Comparing base (0a19823) to head (6fbf718).
Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
.../transport/ReprovisionWorkflowTransportAction.java	9.09%	8 Missing and 2 partials ⚠️
...rk/transport/ProvisionWorkflowTransportAction.java	30.00%	5 Missing and 2 partials ⚠️
.../transport/DeprovisionWorkflowTransportAction.java	33.33%	1 Missing and 3 partials ⚠️
...ensearch/flowframework/util/TenantAwareHelper.java	97.05%	0 Missing and 1 partial ⚠️

❌ Your patch status has failed because the patch coverage (69.01%) is below the target coverage (70.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #1074      +/-   ##
============================================
+ Coverage     75.57%   76.32%   +0.75%     
- Complexity     1050     1075      +25     
============================================
  Files           101      101              
  Lines          5212     5276      +64     
  Branches        498      503       +5     
============================================
+ Hits           3939     4027      +88     
+ Misses         1035     1001      -34     
- Partials        238      248      +10     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[dbwiddis]

tests failing because of jar hell caused by opensearch-project/ml-commons#3475 cc @owaiskazi19

[owaiskazi19]

Looks good overall with comments

[dbwiddis]

@owaiskazi19 @amitgalitz Thanks for your review!

I've addressed some comments in 6fbf718:

• used computeIfPresent() to remove from the map when it hits 0
• used ActionListener.runBefore() in the deprovision case; it is a lot simpler

I'd like to handle changing the value of settings/limits and defaults in another PR (followed by a discussion issue).

I'll consider refactoring all of the multitenant ITs at a later date.

Other comments I've responded that I don't think changes are needed; I've added some comments in code explaining why. In particular I don't think there's a easy way to replace the increment/rollback logic with compareAndSet without really overcomplicating things.

[owaiskazi19]

Thanks for addressing the comments