Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Pull-based Ingestion] add basic NodeStats metrics #17444

Merged
merged 5 commits into from
Mar 3, 2025

Conversation

xuxiong1
Copy link
Contributor

Description

This PR adds the basic NodeStats metrics to pull-based ingestion.

  • Added total polled and total processed message count metrics, as part of IndexShardStats.
  • Tested in the Kafka integration test
  • TODO: add metrics for error handling and consume lag.

Related Issues

Resolves #17077

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@github-actions github-actions bot added enhancement Enhancement or improvement to existing feature or request Indexing Indexing, Bulk Indexing and anything related to indexing labels Feb 24, 2025
Copy link
Contributor

❌ Gradle check result for a3eeb6a: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Copy link
Contributor

❌ Gradle check result for e9ab1ee: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Copy link
Contributor

❌ Gradle check result for 8ddd871: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Copy link
Contributor

❌ Gradle check result for 47c124b: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Copy link
Contributor

❌ Gradle check result for 45f680d: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

xuxiong1 added 5 commits March 3, 2025 11:00
Signed-off-by: xuxiong1 <xiongxug@outlook.com>

# Conflicts:
#	plugins/ingestion-kafka/src/internalClusterTest/java/org/opensearch/plugin/kafka/IngestFromKafkaIT.java
#	server/src/main/java/org/opensearch/index/engine/IngestionEngine.java
Signed-off-by: xuxiong1 <xiongxug@outlook.com>
Signed-off-by: xuxiong1 <xiongxug@outlook.com>
Signed-off-by: xuxiong1 <xiongxug@outlook.com>
Signed-off-by: xuxiong1 <xiongxug@outlook.com>
Copy link
Contributor

github-actions bot commented Mar 3, 2025

❌ Gradle check result for 17db713: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Copy link
Contributor

github-actions bot commented Mar 3, 2025

✅ Gradle check result for 17db713: SUCCESS

Copy link

codecov bot commented Mar 3, 2025

Codecov Report

Attention: Patch coverage is 71.76471% with 24 lines in your changes missing coverage. Please review.

Project coverage is 72.45%. Comparing base (0ffed5e) to head (17db713).
Report is 10 commits behind head on main.

Files with missing lines Patch % Lines
...arch/indices/pollingingest/PollingIngestStats.java 78.57% 3 Missing and 9 partials ⚠️
...rch/indices/pollingingest/DefaultStreamPoller.java 33.33% 4 Missing ⚠️
...nsearch/action/admin/indices/stats/ShardStats.java 62.50% 2 Missing and 1 partial ⚠️
...in/java/org/opensearch/indices/IndicesService.java 0.00% 2 Missing ⚠️
...min/cluster/stats/TransportClusterStatsAction.java 50.00% 1 Missing ⚠️
...a/org/opensearch/index/engine/IngestionEngine.java 0.00% 1 Missing ⚠️
...ndices/pollingingest/MessageProcessorRunnable.java 66.66% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main   #17444      +/-   ##
============================================
+ Coverage     72.42%   72.45%   +0.02%     
- Complexity    65611    65645      +34     
============================================
  Files          5304     5307       +3     
  Lines        304743   304678      -65     
  Branches      44189    44174      -15     
============================================
+ Hits         220701   220743      +42     
+ Misses        65888    65881       -7     
+ Partials      18154    18054     -100     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@andrross andrross merged commit 21f69ca into opensearch-project:main Mar 3, 2025
34 of 36 checks passed
mayanksharma27 pushed a commit to mayanksharma27/OpenSearch that referenced this pull request Mar 5, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Enhancement or improvement to existing feature or request Indexing Indexing, Bulk Indexing and anything related to indexing skip-changelog
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Feature Request] Metrics for pull-based ingestion
4 participants