[Bug]: Performance Regression in OpenSearch 2.19.1 with OpenSearch Benchmark 1.12.0 (Not Observed with 1.11.0)

[prudhvigodithi]

Describe the bug

I've observed a significant performance drop in OpenSearch 2.19.1 when tested with OpenSearch Benchmark 1.12.0. However, the same OpenSearch version (2.19.1) performs consistent with benchmark 1.11.0 in par with OS 2.19.0.

Coming from opensearch-project/OpenSearch#17404 to fix the regression in desc_sort_timestamp for 3.0.0, I have taken 2.19.1 as baseline, but when used benchmark version 1.12.0 I can see a Performance Regression but does not see when used benchmark 1.11.0.

I was able to get this issue consistently on 5 Ec2 instances.

1 segment

{"_shards":{"total":1,"successful":1,"failed":0},"indices":{"big5":{"shards":{"0":[{"routing":{"state":"STARTED","primary":true,"node":"ZfzMd5imR0SMl0jC-2UkYA"},"num_committed_segments":1,"num_search_segments":1,"segments":{"_f1":{"generation":541,"num_docs":116000000,"deleted_docs":0,"size_in_bytes":25901850984,"memory_in_bytes":0,"committed":true,"search":true,"version":"9.12.1","compound":false,"attributes":{"Lucene90StoredFieldsFormat.mode":"BEST_COMPRESSION"}}}}]}}}}


|                                                  Segment count |                     |           1 |        |
|                                                 Min Throughput | desc_sort_timestamp |        0.43 |  ops/s |
|                                                Mean Throughput | desc_sort_timestamp |        0.43 |  ops/s |
|                                              Median Throughput | desc_sort_timestamp |        0.43 |  ops/s |
|                                                 Max Throughput | desc_sort_timestamp |        0.43 |  ops/s |
|                                        50th percentile latency | desc_sort_timestamp |      454100 |     ms |
|                                        90th percentile latency | desc_sort_timestamp |      525790 |     ms |
|                                        99th percentile latency | desc_sort_timestamp |      541906 |     ms |
|                                       100th percentile latency | desc_sort_timestamp |      543706 |     ms |
|                                   50th percentile service time | desc_sort_timestamp |     2303.61 |     ms |
|                                   90th percentile service time | desc_sort_timestamp |     2312.28 |     ms |
|                                   99th percentile service time | desc_sort_timestamp |     2420.03 |     ms |
|                                  100th percentile service time | desc_sort_timestamp |     2420.87 |     ms |
|                                                     error rate | desc_sort_timestamp |           0 |      % |

5 segments

{"_shards":{"total":1,"successful":1,"failed":0},"indices":{"big5":{"shards":{"0":[{"routing":{"state":"STARTED","primary":true,"node":"ZfzMd5imR0SMl0jC-2UkYA"},"num_committed_segments":5,"num_search_segments":5,"segments":{"_3q":{"generation":134,"num_docs":23761000,"deleted_docs":0,"size_in_bytes":5301017262,"memory_in_bytes":0,"committed":true,"search":true,"version":"9.12.1","compound":false,"attributes":{"Lucene90StoredFieldsFormat.mode":"BEST_COMPRESSION"}},"_av":{"generation":391,"num_docs":20250288,"deleted_docs":0,"size_in_bytes":4519365696,"memory_in_bytes":0,"committed":true,"search":true,"version":"9.12.1","compound":false,"attributes":{"Lucene90StoredFieldsFormat.mode":"BEST_COMPRESSION"}},"_c8":{"generation":440,"num_docs":23351854,"deleted_docs":0,"size_in_bytes":5208262447,"memory_in_bytes":0,"committed":true,"search":true,"version":"9.12.1","compound":false,"attributes":{"Lucene90StoredFieldsFormat.mode":"BEST_COMPRESSION"}},"_f0":{"generation":540,"num_docs":23160728,"deleted_docs":0,"size_in_bytes":5167648144,"memory_in_bytes":0,"committed":true,"search":true,"version":"9.12.1","compound":false,"attributes":{"Lucene90StoredFieldsFormat.mode":"BEST_COMPRESSION"}},"_f1":{"generation":541,"num_docs":25476130,"deleted_docs":0,"size_in_bytes":5679438952,"memory_in_bytes":0,"committed":true,"search":true,"version":"9.12.1","compound":false,"attributes":{"Lucene90StoredFieldsFormat.mode":"BEST_COMPRESSION"}}}}]}}}}

|                                            Heap used for norms |                     |           0 |     MB |
|                                           Heap used for points |                     |           0 |     MB |
|                                    Heap used for stored fields |                     |           0 |     MB |
|                                                  Segment count |                     |           5 |        |
|                                                 Min Throughput | desc_sort_timestamp |           2 |  ops/s |
|                                                Mean Throughput | desc_sort_timestamp |           2 |  ops/s |
|                                              Median Throughput | desc_sort_timestamp |           2 |  ops/s |
|                                                 Max Throughput | desc_sort_timestamp |           2 |  ops/s |
|                                        50th percentile latency | desc_sort_timestamp |     53.5814 |     ms |
|                                        90th percentile latency | desc_sort_timestamp |     54.8334 |     ms |
|                                        99th percentile latency | desc_sort_timestamp |     63.0936 |     ms |
|                                       100th percentile latency | desc_sort_timestamp |     66.8695 |     ms |
|                                   50th percentile service time | desc_sort_timestamp |     52.2557 |     ms |
|                                   90th percentile service time | desc_sort_timestamp |     53.1345 |     ms |
|                                   99th percentile service time | desc_sort_timestamp |     61.8123 |     ms |
|                                  100th percentile service time | desc_sort_timestamp |     65.1388 |     ms |
|                                                     error rate | desc_sort_timestamp |           0 |      % |


10 segments

{"_shards":{"total":1,"successful":1,"failed":0},"indices":{"big5":{"shards":{"0":[{"routing":{"state":"STARTED","primary":true,"node":"ZfzMd5imR0SMl0jC-2UkYA"},"num_committed_segments":10,"num_search_segments":10,"segments":{"_3q":{"generation":134,"num_docs":23761000,"deleted_docs":0,"size_in_bytes":5301017262,"memory_in_bytes":0,"committed":true,"search":true,"version":"9.12.1","compound":false,"attributes":{"Lucene90StoredFieldsFormat.mode":"BEST_COMPRESSION"}},"_av":{"generation":391,"num_docs":20250288,"deleted_docs":0,"size_in_bytes":4519365696,"memory_in_bytes":0,"committed":true,"search":true,"version":"9.12.1","compound":false,"attributes":{"Lucene90StoredFieldsFormat.mode":"BEST_COMPRESSION"}},"_c8":{"generation":440,"num_docs":23351854,"deleted_docs":0,"size_in_bytes":5208262447,"memory_in_bytes":0,"committed":true,"search":true,"version":"9.12.1","compound":false,"attributes":{"Lucene90StoredFieldsFormat.mode":"BEST_COMPRESSION"}},"_cd":{"generation":445,"num_docs":4463503,"deleted_docs":0,"size_in_bytes":984515846,"memory_in_bytes":0,"committed":true,"search":true,"version":"9.12.1","compound":true,"attributes":{"Lucene90StoredFieldsFormat.mode":"BEST_COMPRESSION"}},"_cz":{"generation":467,"num_docs":4541810,"deleted_docs":0,"size_in_bytes":1001614039,"memory_in_bytes":0,"committed":true,"search":true,"version":"9.12.1","compound":true,"attributes":{"Lucene90StoredFieldsFormat.mode":"BEST_COMPRESSION"}},"_dm":{"generation":490,"num_docs":4815444,"deleted_docs":0,"size_in_bytes":1061976308,"memory_in_bytes":0,"committed":true,"search":true,"version":"9.12.1","compound":true,"attributes":{"Lucene90StoredFieldsFormat.mode":"BEST_COMPRESSION"}},"_e5":{"generation":509,"num_docs":4426373,"deleted_docs":0,"size_in_bytes":976329628,"memory_in_bytes":0,"committed":true,"search":true,"version":"9.12.1","compound":true,"attributes":{"Lucene90StoredFieldsFormat.mode":"BEST_COMPRESSION"}},"_er":{"generation":531,"num_docs":4862630,"deleted_docs":0,"size_in_bytes":1073347466,"memory_in_bytes":0,"committed":true,"search":true,"version":"9.12.1","compound":true,"attributes":{"Lucene90StoredFieldsFormat.mode":"BEST_COMPRESSION"}},"_ez":{"generation":539,"num_docs":2366370,"deleted_docs":0,"size_in_bytes":523776577,"memory_in_bytes":0,"committed":true,"search":true,"version":"9.12.1","compound":true,"attributes":{"Lucene90StoredFieldsFormat.mode":"BEST_COMPRESSION"}},"_f0":{"generation":540,"num_docs":23160728,"deleted_docs":0,"size_in_bytes":5167648144,"memory_in_bytes":0,"committed":true,"search":true,"version":"9.12.1","compound":false,"attributes":{"Lucene90StoredFieldsFormat.mode":"BEST_COMPRESSION"}}}}]}}}}

|                                                  Segment count |                     |          10 |        |
|                                                 Min Throughput | desc_sort_timestamp |           2 |  ops/s |
|                                                Mean Throughput | desc_sort_timestamp |           2 |  ops/s |
|                                              Median Throughput | desc_sort_timestamp |           2 |  ops/s |
|                                                 Max Throughput | desc_sort_timestamp |        2.01 |  ops/s |
|                                        50th percentile latency | desc_sort_timestamp |     41.3232 |     ms |
|                                        90th percentile latency | desc_sort_timestamp |     42.4959 |     ms |
|                                        99th percentile latency | desc_sort_timestamp |     50.1217 |     ms |
|                                       100th percentile latency | desc_sort_timestamp |     53.8461 |     ms |
|                                   50th percentile service time | desc_sort_timestamp |     39.9178 |     ms |
|                                   90th percentile service time | desc_sort_timestamp |      41.307 |     ms |
|                                   99th percentile service time | desc_sort_timestamp |      48.512 |     ms |
|                                  100th percentile service time | desc_sort_timestamp |     52.3414 |     ms |
|                                                     error rate | desc_sort_timestamp |           0 |      % |

To reproduce

• Created an r5.xlarge Ec2 ami-05b10e08d247fb927. Using EBS Gp3 16000 IOPS, 500Gb and 1000 throughput. I have seen the same results with EBS defaults 3000 IOPS and 125 throughput, so increasing dint help.

• Install java sudo yum install -y https://corretto.aws/downloads/latest/amazon-corretto-21-x64-linux-jdk.rpm.

• sudo yum install -y pip.

• pip install opensearch-benchmark which installs the latest 1.12.0.

opensearch-benchmark --version
opensearch-benchmark 1.12.0

• sudo yum install -y git.

• Download OS 2.19.1 with wget https://artifacts.opensearch.org/releases/core/opensearch/2.19.1/opensearch-min-2.19.1-linux-x64.tar.gz.

• Update the bash profile to use jdk21 as default.

vi ~/.bashrc
export JAVA_HOME="/usr/lib/jvm/java-21-amazon-corretto"
export PATH=$JAVA_HOME/bin:$PATH
source ~/.bashrc

java --version
openjdk 21.0.6 2025-01-21 LTS
OpenJDK Runtime Environment Corretto-21.0.6.7.1 (build 21.0.6+7-LTS)
OpenJDK 64-Bit Server VM Corretto-21.0.6.7.1 (build 21.0.6+7-LTS, mixed mode, sharing)

• mkdir DataDir and update the opensearch-2.19.1/config/opensearch.yml to use this directory. Example

path.data: /home/ec2-user/OS_2191/DataDir
discovery.type: single-node

• Update the opensearch-2.19.1/config/jvm.options

-Xms4g
-Xmx4g

• Coming from https://opensearch.org/docs/latest/getting-started/quickstart/ , disabled memory paging and swapping sudo swapoff -a and updated the /etc/sysctl.conf to vm.max_map_count=262144.

• Ensure the cluster is up and running

{
  "name" : "ip-172-31-32-21.ec2.internal",
  "cluster_name" : "opensearch",
  "cluster_uuid" : "T5lvPPyTQgymJunlb5Qm7Q",
  "version" : {
    "distribution" : "opensearch",
    "number" : "2.19.1",
    "build_type" : "tar",
    "build_hash" : "2e4741fb45d1b150aaeeadf66d41445b23ff5982",
    "build_date" : "2025-02-27T01:16:47.726162386Z",
    "build_snapshot" : false,
    "lucene_version" : "9.12.1",
    "minimum_wire_compatibility_version" : "7.10.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

• Once the cluster is up and running, execute the cli

opensearch-benchmark execute-test --target-hosts http://127.0.0.1:9200 --workload big5 --client-options timeout:120 --workload-params="number_of_shards:1,bulk_indexing_clients:1,number_of_replicas:0" --kill-running-processes --include-tasks delete-index,create-index,check-cluster-health,index-append,refresh-after-index,force-merge,refresh-after-force-merge,wait-until-merges-finish 

• Once the data is indexed run opensearch-benchmark execute-test --target-hosts http://127.0.0.1:9200 --workload big5 --client-options timeout:120 --kill-running-processes --include-tasks desc_sort_timestamp.

Expected behavior

With opensearch-benchmark 1.11.0.

With default created 23 segments 

|                                       Heap used for doc values |                     |           0 |     MB |
|                                            Heap used for terms |                     |           0 |     MB |
|                                            Heap used for norms |                     |           0 |     MB |
|                                           Heap used for points |                     |           0 |     MB |
|                                    Heap used for stored fields |                     |           0 |     MB |
|                                                  Segment count |                     |          23 |        |
|                                                 Min Throughput | desc_sort_timestamp |           2 |  ops/s |
|                                                Mean Throughput | desc_sort_timestamp |        2.01 |  ops/s |
|                                              Median Throughput | desc_sort_timestamp |        2.01 |  ops/s |
|                                                 Max Throughput | desc_sort_timestamp |        2.01 |  ops/s |
|                                        50th percentile latency | desc_sort_timestamp |     9.39446 |     ms |
|                                        90th percentile latency | desc_sort_timestamp |     9.81926 |     ms |
|                                        99th percentile latency | desc_sort_timestamp |     12.4212 |     ms |
|                                       100th percentile latency | desc_sort_timestamp |     12.9212 |     ms |
|                                   50th percentile service time | desc_sort_timestamp |      8.0011 |     ms |
|                                   90th percentile service time | desc_sort_timestamp |     8.43834 |     ms |
|                                   99th percentile service time | desc_sort_timestamp |     11.2254 |     ms |
|                                  100th percentile service time | desc_sort_timestamp |      11.282 |     ms |
|                                                     error rate | desc_sort_timestamp |           0 |      % |


Force merged from 23 to 10 segments.

|                                                  Segment count |                     |          10 |        |
|                                                 Min Throughput | desc_sort_timestamp |           2 |  ops/s |
|                                                Mean Throughput | desc_sort_timestamp |        2.01 |  ops/s |
|                                              Median Throughput | desc_sort_timestamp |           2 |  ops/s |
|                                                 Max Throughput | desc_sort_timestamp |        2.01 |  ops/s |
|                                        50th percentile latency | desc_sort_timestamp |     8.81708 |     ms |
|                                        90th percentile latency | desc_sort_timestamp |      9.6414 |     ms |
|                                        99th percentile latency | desc_sort_timestamp |     10.8075 |     ms |
|                                       100th percentile latency | desc_sort_timestamp |     10.9072 |     ms |
|                                   50th percentile service time | desc_sort_timestamp |     7.42113 |     ms |
|                                   90th percentile service time | desc_sort_timestamp |     8.17049 |     ms |
|                                   99th percentile service time | desc_sort_timestamp |     9.51799 |     ms |
|                                  100th percentile service time | desc_sort_timestamp |     9.54889 |     ms |
|                                                     error rate | desc_sort_timestamp |           0 |      % |

Force merged from 23 to 5 segments.


                                                  Segment count |                     |           5 |        |
|                                                 Min Throughput | desc_sort_timestamp |           2 |  ops/s |
|                                                Mean Throughput | desc_sort_timestamp |        2.01 |  ops/s |
|                                              Median Throughput | desc_sort_timestamp |        2.01 |  ops/s |
|                                                 Max Throughput | desc_sort_timestamp |        2.01 |  ops/s |
|                                        50th percentile latency | desc_sort_timestamp |     10.5416 |     ms |
|                                        90th percentile latency | desc_sort_timestamp |     11.6064 |     ms |
|                                        99th percentile latency | desc_sort_timestamp |     29.9041 |     ms |
|                                       100th percentile latency | desc_sort_timestamp |      30.757 |     ms |
|                                   50th percentile service time | desc_sort_timestamp |     9.14574 |     ms |
|                                   90th percentile service time | desc_sort_timestamp |      10.499 |     ms |
|                                   99th percentile service time | desc_sort_timestamp |     28.9229 |     ms |
|                                  100th percentile service time | desc_sort_timestamp |     28.9932 |     ms |
|                                                     error rate | desc_sort_timestamp |           0 |      % |



Force merged from 23 to 1 segments.

|                                    Heap used for stored fields |                     |           0 |     MB |
|                                                  Segment count |                     |           1 |        |
|                                                 Min Throughput | desc_sort_timestamp |           2 |  ops/s |
|                                                Mean Throughput | desc_sort_timestamp |        2.01 |  ops/s |
|                                              Median Throughput | desc_sort_timestamp |        2.01 |  ops/s |
|                                                 Max Throughput | desc_sort_timestamp |        2.01 |  ops/s |
|                                        50th percentile latency | desc_sort_timestamp |     13.4382 |     ms |
|                                        90th percentile latency | desc_sort_timestamp |     14.6247 |     ms |
|                                        99th percentile latency | desc_sort_timestamp |     18.2348 |     ms |
|                                       100th percentile latency | desc_sort_timestamp |     96.1659 |     ms |
|                                   50th percentile service time | desc_sort_timestamp |     12.1124 |     ms |
|                                   90th percentile service time | desc_sort_timestamp |     13.1238 |     ms |
|                                   99th percentile service time | desc_sort_timestamp |     16.6703 |     ms |
|                                  100th percentile service time | desc_sort_timestamp |     95.2705 |     ms |
|                                                     error rate | desc_sort_timestamp |           0 |      % |

Additional context

• To debug further I have shared the data dir (DataDir) which has the big5 index data created by benchmark with @expani and running benchmark cli on his host shows the same issue.

[prudhvigodithi]

Adding @rishabh6788 @gkamat to please take a look.
Thanks
@expani @getsaurabh02 @mch2

[rishabh6788]

Created a single node cluster on r5.xlarge host, configured to use 4G heap. Ingested data using below command:

opensearch-benchmark execute-test  --workload=big5 --target-hosts=opense-clust-5KBIuufon6Qp-5937377bbdb4b4e7.elb.us-east-1.amazonaws.com:80 --workload-params '{"number_of_shards":1,"number_of_replicas":0,"bulk_indexing_clients":1}' --include-tasks delete-index,create-index,check-cluster-health,index-append,refresh-after-index,force-merge,refresh-after-force-merge,wait-until-merges-finish

OSB version is 1.12:

[ec2-user@ip-172-31-173-178 ~]$ opensearch-benchmark --versionopensearch-benchmark 1.12.0

desc_sort_result with default created 21 segments:
opensearch-benchmark execute-test --workload=big5 --target-hosts=opense-clust-5KBIuufon6Qp-5937377bbdb4b4e7.elb.us-east-1.amazonaws.com:80 --include-tasks desc_sort_timestamp

|                                                  Segment count |                     |          22 |        |
|                                                 Min Throughput | desc_sort_timestamp |        2.01 |  ops/s |
|                                                Mean Throughput | desc_sort_timestamp |        2.01 |  ops/s |
|                                              Median Throughput | desc_sort_timestamp |        2.01 |  ops/s |
|                                                 Max Throughput | desc_sort_timestamp |        2.01 |  ops/s |
|                                        50th percentile latency | desc_sort_timestamp |     11.6608 |     ms |
|                                        90th percentile latency | desc_sort_timestamp |     12.1544 |     ms |
|                                        99th percentile latency | desc_sort_timestamp |      14.379 |     ms |
|                                       100th percentile latency | desc_sort_timestamp |     16.8428 |     ms |
|                                   50th percentile service time | desc_sort_timestamp |     10.3487 |     ms |
|                                   90th percentile service time | desc_sort_timestamp |     10.8433 |     ms |
|                                   99th percentile service time | desc_sort_timestamp |     13.2102 |     ms |
|                                  100th percentile service time | desc_sort_timestamp |      15.507 |     ms |
|                                                     error rate | desc_sort_timestamp |           0 |      % |

Merged down to 10 segments:
opensearch-benchmark execute-test --workload=big5 --target-hosts=opense-clust-5KBIuufon6Qp-5937377bbdb4b4e7.elb.us-east-1.amazonaws.com:80 --include-tasks desc_sort_timestamp

|                                                  Segment count |                     |          12 |        |
|                                                 Min Throughput | desc_sort_timestamp |        2.01 |  ops/s |
|                                                Mean Throughput | desc_sort_timestamp |        2.01 |  ops/s |
|                                              Median Throughput | desc_sort_timestamp |        2.01 |  ops/s |
|                                                 Max Throughput | desc_sort_timestamp |        2.01 |  ops/s |
|                                        50th percentile latency | desc_sort_timestamp |     8.13745 |     ms |
|                                        90th percentile latency | desc_sort_timestamp |     8.78359 |     ms |
|                                        99th percentile latency | desc_sort_timestamp |     14.4368 |     ms |
|                                       100th percentile latency | desc_sort_timestamp |     14.8832 |     ms |
|                                   50th percentile service time | desc_sort_timestamp |     6.98082 |     ms |
|                                   90th percentile service time | desc_sort_timestamp |     7.18361 |     ms |
|                                   99th percentile service time | desc_sort_timestamp |     13.1606 |     ms |
|                                  100th percentile service time | desc_sort_timestamp |     13.5472 |     ms |
|                                                     error rate | desc_sort_timestamp |           0 |      % |

It is more or less in line with 1.11 numbers.
The logic for preparing benchmark and bulk indexing the data has not changed, there could be 2 reasons for you to be getting higher than normal latencies

• noisy neighbor issue due to OSB and OS running on the same node.
• The overall data alignment at segment level. We have seen variations of 100s of magnitude, specifically in sort queries, when we used to do full ingestion and search in single run. @IanHoang did a deep dive on it and it basically boiled down to how documents were finally aligned at segment level, which OSB cannot control.

[gkamat]

@rishabh6788 has very good points above. One way to confirm whether there is indeed an issue is to install the two OSB versions on two separate (but identical) load-generation instances and target the same cluster in separate query-only tests (after ingestion and segment merge has completed.)

Please try this out and update the issue with your findings. Thanks.