Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adjust SIRI ServiceBus-client parameters to improve performance #5741

Conversation

jtorin
Copy link
Contributor

@jtorin jtorin commented Mar 11, 2024

Summary

Adjust SIRI ServiceBus-client parameters to improve performance

Issue

  • Move to ServiceBusReceiveMode.RECEIVE_AND_DELETE instead of PEEK_LOCK. In this mode the message is deleted from the subscription as soon as it has been delivered. In case of an error this means that the message is lost but 1) the data is applied to the graph in another thread anyway and 2) there is nothing that indicates that a second attempt on the same message data would be more successful.
  • Fetch more than one message at a time, greatly improving message throughput even with small values. Roundtrip latency of the message fetching can in some situations otherwise lead to the client falling behind. Add config parameter 'prefetchCount' with a default value of 10 messages.
  • Lower the AutoDeleteOnIdle of subscriptions from one day to default one hour, still a conservative value. Despite having automatic removal of subscriptions on shutdown, lingering subscriptions may still happen, for example if the JVM dies. Add config parameter 'autoDeleteOnIdle' with a default value of one hour.

Testing

Have been running in production for a few months at Skånetrafiken.

@jtorin jtorin requested a review from a team as a code owner March 11, 2024 13:44
@leonardehrenfried
Copy link
Member

You need to regenerate the documentation

* Move to ServiceBusReceiveMode.RECEIVE_AND_DELETE instead of PEEK_LOCK.
  In this mode the message is deleted from the subscription as soon as
  it has been delivered. In case of an error this means that the message
  is lost but 1) the data is applied to the graph in another thread anyway
  and 2) there is nothing that indicates that a second attempt on the same
  message data would be more successful.
* Fetch more than one message at a time, greatly improving message throughput
  even with small values. Roundtrip latency of the message fetching can
  in some situations otherwise lead to the client falling behind.
  Add config parameter 'prefetchCount' with a default value of 10 messages.
* Lower the AutoDeleteOnIdle of subscriptions from one day to default one hour,
  still a conservative value. Despite having automatic removal of subscriptions on
  shutdown, lingering subscriptions may still happen, for example if the JVM dies.
  Add config parameter 'autoDeleteOnIdle' with a default value of one hour.
@jtorin jtorin force-pushed the adjust-siri-servicebus-client-parameters branch from e924f50 to 163b0aa Compare March 11, 2024 18:31
Copy link

codecov bot commented Mar 11, 2024

Codecov Report

Attention: Patch coverage is 63.63636% with 8 lines in your changes are missing coverage. Please review.

Project coverage is 67.77%. Comparing base (0532203) to head (163b0aa).
Report is 13 commits behind head on dev-2.x.

Files Patch % Lines
...t/siri/updater/azure/AbstractAzureSiriUpdater.java 0.00% 6 Missing ⚠️
...siri/updater/azure/SiriAzureUpdaterParameters.java 66.66% 2 Missing ⚠️
Additional details and impacted files
@@              Coverage Diff              @@
##             dev-2.x    #5741      +/-   ##
=============================================
- Coverage      67.77%   67.77%   -0.01%     
+ Complexity     16474    16473       -1     
=============================================
  Files           1901     1901              
  Lines          72121    72142      +21     
  Branches        7430     7430              
=============================================
+ Hits           48881    48892      +11     
- Misses         20727    20736       +9     
- Partials        2513     2514       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@t2gran t2gran modified the milestones: 2.5, 2.6 (next release) Mar 13, 2024
@jtorin jtorin merged commit 5edc5ae into opentripplanner:dev-2.x Mar 14, 2024
5 checks passed
@jtorin jtorin deleted the adjust-siri-servicebus-client-parameters branch March 14, 2024 09:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants