[BUG] Increased latency for Keyword Term operations in Big5

[expani]

Describe the bug

We have seen an increase in latency ~28% for keyword-terms and keyword-terms-low-cardinality in Big5 Workload b/w OS 2.19 and OS 3.0

OS 3.0 * OS 2.19 indicates that the OpenSearch server was running 3.0 whereas the index used was created in OS 2.19. This was done to eliminate any suspicion of the bug arising from an indexing change in Lucene.

Keyword Terms

Metric Name	Operation/Query name	OS 2.19	OS 3.0	OS 3.0 * OS 2.19	Unit
50th percentile latency	keyword-terms	46.3265	64.1604	65.1848	ms
90th percentile latency	keyword-terms	47.0906	64.7056	65.734	ms
99th percentile latency	keyword-terms	53.7872	74.5766	68.4211	ms
100th percentile latency	keyword-terms	59.4091	77.1213	69.5411	ms
50th percentile service time	keyword-terms	45.0249	62.8199	63.9045	ms
90th percentile service time	keyword-terms	45.6936	63.229	64.3549	ms
99th percentile service time	keyword-terms	52.9548	72.8261	67.0538	ms
100th percentile service time	keyword-terms	57.6884	76.0182	68.5125	ms

Keyword Terms Low Cardinality

Metric Name	Operation/Query name	OS 2.19	OS 3.0	OS 3.0 * OS 2.19	Unit
50th percentile latency	keyword-terms-low-cardinality	39.6217	58.8665	59.9801	ms
90th percentile latency	keyword-terms-low-cardinality	40.119	59.5365	60.5104	ms
99th percentile latency	keyword-terms-low-cardinality	41.2324	73.0814	62.4343	ms
100th percentile latency	keyword-terms-low-cardinality	68.1012	79.1284	63.2587	ms
50th percentile service time	keyword-terms-low-cardinality	38.301	57.5659	58.6619	ms
90th percentile service time	keyword-terms-low-cardinality	38.6461	57.9068	59.1032	ms
99th percentile service time	keyword-terms-low-cardinality	39.8793	72.1398	61.1524	ms
100th percentile service time	keyword-terms-low-cardinality	67.2168	77.6797	61.5098	ms

Related component

Search:Performance

To Reproduce

Run the keyword-terms operation in Big5 workload and compare the latencies b/w OS 2.19 and mainline.

Expected behavior

The latency should be same or reduced as OS 2.19

Additional Details

Meta