Process seeding of split fate operations in batches #5404
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Updates the Seeder in the Manager that handles seeding split fate ops to use a single thread and to submit multiple outstanding operations to be seeded together instead of individually in order to improve performance. The user fate store will now track outstanding fate operations and return a future for each pending operation that will be completed when the batch is submitted. This closes apache#5160
|
I didn't add any new tests for this (at least not yet) because I figured the existing ITs would catch any issues. I'm going to kick off a full IT build now. Also, I deprecated the |
keith-turner
approved these changes
Mar 17, 2025
| @@ -461,6 +461,7 @@ public enum Property { | |||
| + "indefinitely. Default is 0 to block indefinitely. Only valid when tserver available " | |||
| + "threshold is set greater than 0.", | |||
| "1.10.0"), | |||
| @Deprecated(since = "4.0.0") | |||
There was a problem hiding this comment.
This property can be dropped instead of deprecated as it was introduced in main.
| // to a conditional writer and any known results will be removed | ||
| // from the pending map. Unknown results will be re-attempted up | ||
| // to the maxAttempts count | ||
| for (int attempt = 0; attempt < maxAttempts; attempt++) { |
There was a problem hiding this comment.
Seems like this will continue to loop even after pending is empty, could add a check for that.
Suggested change
| for (int attempt = 0; attempt < maxAttempts; attempt++) { | |
| for (int attempt = 0; attempt < maxAttempts && !pending.isEmpty(); attempt++) { |
There was a problem hiding this comment.
Good catch, i'll fix that
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Updates the Seeder in the Manager that handles seeding split fate ops to use a single thread and to submit multiple outstanding operations to be seeded together instead of individually in order to improve performance. The user fate store will now track outstanding fate operations and return a future for each pending operation that will be completed when the batch is submitted.
This closes #5160