Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Log iteration to result count in BulkImport code #5312

Merged
merged 6 commits into from
Feb 25, 2025
Merged

Log iteration to result count in BulkImport code #5312

Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
e1e41ad
Log iteration to result count in BulkImport code
DomGarguilo Feb 6, 2025
ccb2b4e
move log message to correct place
DomGarguilo Feb 10, 2025
7ee6987
Merge remote-tracking branch 'upstream/2.1' into bulkImportScalingBug
DomGarguilo Feb 10, 2025
04742d9
Merge remote-tracking branch 'upstream/2.1' into bulkImportScalingBug
DomGarguilo Feb 25, 2025
03de2e4
Log aggregate stats instead of individual calls
DomGarguilo Feb 25, 2025
c651f99
use debug instead of info for log level
DomGarguilo Feb 25, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions core/src/main/java/org/apache/accumulo/core/clientImpl/bulk/BulkImport.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -338,7 +338,10 @@ public static List<KeyExtent> findOverlappingTablets(Function<Text,KeyExtent> ro

List<KeyExtent> result = new ArrayList<>();
Text row = new Text();
long iterationCount = 0;
while (true) {
iterationCount++;

Copy link
Contributor

@keith-turner keith-turner Feb 6, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe this code is ok, it jump around in the metadata table and does not scan over tablets it does not need to.

The code that does scan over unrelated tablets in the metadata table is here . Would be good to track counts of skipped vs total tablets in that code. The function is called multiple times for a single bulk load and could get the counts from all calls and log after that loop that calls the function.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. I moved the log message to the spot you mentioned.

I added a check so things are only logged if there are "wasted iterations". With that check in place I can't get the message to be logged.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a check so things are only logged if there are "wasted iterations". With that check in place I can't get the message to be logged.

Which test are you running? The code seems ok, but it would be good to see the log message.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was running BulkNewIT

row = nextRowFunction.apply(row);
if (row == null) {
break;
Expand All @@ -353,6 +356,8 @@ public static List<KeyExtent> findOverlappingTablets(Function<Text,KeyExtent> ro
}
}

log.debug("performed {} iterations to identify {} overlapping tablets", iterationCount,
result.size());
return result;
}

Expand Down