[BUG] org.opensearch.indices.stats.IndexStatsIT.classMethod is flaky

[lukas-vlcek]

Describe the bug

The test case org.opensearch.indices.stats.IndexStatsIT.classMethod is flaky.

java.lang.RuntimeException: file handle leaks: [FileChannel(/var/jenkins/workspace/gradle-check/search/server/build/testrun/internalClusterTest/temp/org.opensearch.indices.stats.IndexStatsIT_62DE73D5CA8E5066-001/tempDir-002/node_s1/nodes/0/indices/pzfmLv3qSt-NaGZBwT2z8Q/0/index/_f.fdx), FileChannel(/var/jenkins/workspace/gradle-check/search/server/build/testrun/internalClusterTest/temp/org.opensearch.indices.stats.IndexStatsIT_62DE73D5CA8E5066-001/tempDir-002/node_s1/nodes/0/indices/pzfmLv3qSt-NaGZBwT2z8Q/0/index/_f.fdt)]
	at __randomizedtesting.SeedInfo.seed([62DE73D5CA8E5066]:0)
	at org.apache.lucene.tests.mockfile.LeakFS.onClose(LeakFS.java:63)
	at org.apache.lucene.tests.mockfile.FilterFileSystem.close(FilterFileSystem.java:69)
	at org.apache.lucene.tests.mockfile.FilterFileSystem.close(FilterFileSystem.java:70)
	at org.apache.lucene.tests.util.TestRuleTemporaryFilesCleanup.afterAlways(TestRuleTemporaryFilesCleanup.java:223)
	at com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterAlways(TestRuleAdapter.java:31)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.Exception
	at org.apache.lucene.tests.mockfile.LeakFS.onOpen(LeakFS.java:46)
	at org.apache.lucene.tests.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:82)
	at org.apache.lucene.tests.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:202)
	at org.apache.lucene.tests.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:171)
	at java.base/java.nio.channels.FileChannel.open(FileChannel.java:309)
	at java.base/java.nio.channels.FileChannel.open(FileChannel.java:369)
	at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:78)
	at org.opensearch.index.store.FsDirectoryFactory$HybridDirectory.openInput(FsDirectoryFactory.java:183)
	at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:101)
	at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:101)
	at org.apache.lucene.codecs.lucene90.compressing.FieldsIndexReader.<init>(FieldsIndexReader.java:68)
	at org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader.<init>(Lucene90CompressingStoredFieldsReader.java:166)
	at org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsFormat.fieldsReader(Lucene90CompressingStoredFieldsFormat.java:133)
	at org.apache.lucene.codecs.lucene90.Lucene90StoredFieldsFormat.fieldsReader(Lucene90StoredFieldsFormat.java:136)
	at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:139)
	at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:96)
	at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:178)
	at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:220)
	at org.apache.lucene.index.IndexWriter.lambda$getReader$0(IndexWriter.java:542)
	at org.apache.lucene.index.IndexWriter.lambda$getReader$1(IndexWriter.java:632)
	at org.apache.lucene.index.IndexWriter$2.onMergeComplete(IndexWriter.java:3893)
	at org.apache.lucene.index.IndexWriter.commitMerge(IndexWriter.java:4552)
	at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:5432)
	at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4740)
	at org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge(IndexWriter.java:6541)
	at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:639)
	at org.opensearch.index.engine.OpenSearchConcurrentMergeScheduler.doMerge(OpenSearchConcurrentMergeScheduler.java:120)
	at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:700)

All details:
https://build.ci.opensearch.org/job/gradle-check/34698/testReport/org.opensearch.indices.stats/IndexStatsIT/classMethod/

This specific test was mentioned in this ticket in the past as well: #5766 (comment)

Related component

Search

To Reproduce

REPRODUCE WITH:

./gradlew ':server:internalClusterTest' --tests "org.opensearch.indices.stats.IndexStatsIT" -Dtests.seed=62DE73D5CA8E5066 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=en -Dtests.timezone=Etc/UTC -Druntime.java=21
NOTE: test params are: codec=Lucene99, sim=Asserting(RandomSimilarity(queryNorm=true): {}), locale=es-VE, timezone=America/Guadeloupe
NOTE: Linux 5.15.0-1048-aws amd64/Eclipse Adoptium 21.0.2 (64-bit)/cpus=32,threads=1,free=1455338656,total=3221225472
NOTE: All tests run in this JVM: [RepositoryBlocksIT, DataStreamTestCase, BulkProcessorClusterSettingsIT, SimpleBlocksIT, ClusterRerouteIT, DelayedAllocationIT, ClusterDisruptionIT, MetadataNodesIT, WaitUntilRefreshIT, RemoveCorruptedShardDataCommandIT, DedicatedClusterManagerGetFieldMappingIT, SegmentReplicationRelocationIT, IndexStatsIT]

Expected behavior

The test should always pass

Additional Details

Plugins

Set of plugins used during CI tests.

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

CI.

Additional context
Add any other context about the problem here.

[andrross]

[Triage - attendees 1 2 3]
@lukas-vlcek Thanks for filing this issue.

[nisgoel-amazon]

This test is not failing on local, I have ran this test more than 50 times.