[BUG] org.opensearch.snapshots.SegmentReplicationSnapshotIT os flaky

[rajiv-kv]

Describe the bug

SegmentReplicationSnapshotIT gets timed out in build due to OOM errors in test

Build :- https://build.ci.opensearch.org/job/gradle-check/35480/testReport/junit/org.opensearch.snapshots/SegmentReplicationSnapshotIT/

Mac 20, 2024 7:20:46 AM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
WARNING: Uncaught exception in thread: Thread[#7849,opensearch[node_t1][generic][T#1],5,TGRP-SegmentReplicationSnapshotIT]
java.lang.OutOfMemoryError: Java heap space

Mac 20, 2024 7:20:46 AM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
WARNING: Uncaught exception in thread: Thread[#7845,opensearch[node_t1][scheduler][T#1],5,TGRP-SegmentReplicationSnapshotIT]
java.lang.OutOfMemoryError: Java heap space

Mac 20, 2024 7:20:46 AM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
WARNING: Uncaught exception in thread: Thread[#7832,opensearch[node_t0][scheduler][T#1],5,TGRP-SegmentReplicationSnapshotIT]
java.lang.OutOfMemoryError: Java heap space

Related component

Build

To Reproduce

./gradlew ':server:internalClusterTest' --tests "org.opensearch.snapshots.SegmentReplicationSnapshotIT" -Dtests.seed=B5E2BB78064B6241 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=en -Dtests.timezone=Etc/UTC -Druntime.java=21

Expected behavior

Test should pass always

Additional Details

Plugins
Please list all plugins currently enabled.

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: [e.g. iOS]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.