Fix flaky IT test on range query in RestMLInferenceSearchRequestProcessorIT

[mingshl]

Description

Earlier the RestMLInferenceSearchRequestProcessorIT has an IT test rewrites to a range query searching on a keyword field, it's flaky, so change to querying an integer field to return stable search result.

Check List

• New functionality includes testing.
• New functionality has been documented.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[mingshl]

flaky test not related to this change, rerun

Suite: Test class org.opensearch.ml.rest.RestCohereInferenceIT
  2> REPRODUCE WITH: ./gradlew ':opensearch-ml-plugin:integTest' --tests "org.opensearch.ml.rest.RestCohereInferenceIT.test_cohereInference_withDifferent_postProcessFunction" -Dtests.seed=6A06CFD3516A4040 -Dtests.security.manager=false -Dtests.locale=sv-SE -Dtests.timezone=America/Cayman -Druntime.java=21
  2> java.lang.AssertionError: failed to run test with test name: connector.post_process.cohere_v2.embedding.ubinary_test
        at __randomizedtesting.SeedInfo.seed([6A06CFD3516A4040:998857A176683568]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.assertTrue(Assert.java:42)
        at org.opensearch.ml.rest.RestCohereInferenceIT.validateOutput(RestCohereInferenceIT.java:85)
        at org.opensearch.ml.rest.RestCohereInferenceIT.test_cohereInference_withDifferent_postProcessFunction(RestCohereInferenceIT.java:74)
  2> NOTE: leaving temporary files on disk at: /__w/ml-commons/ml-commons/plugin/build/testrun/integTest/temp/org.opensearch.ml.rest.RestCohereInferenceIT_6A06CFD3516A4040-001
  2> NOTE: test params are: codec=Asserting(Lucene101): {}, docValues:{}, maxPointsInLeafNode=386, maxMBSortInHeap=6.152622036236546, sim=Asserting(RandomSimilarity(queryNorm=false): {}), locale=sv-SE, timezone=America/Cayman
  2> NOTE: Linux 6.8.0-1021-azure amd64/Azul Systems, Inc. 21.0.6 (64-bit)/cpus=4,threads=1,free=383929800,total=536870912
  2> NOTE: All tests run in this JVM: [MLModelAutoReDeployerIT, RestBedRockInferenceIT, RestCohereInferenceIT]
    NOTE: test params are: codec=Asserting(Lucene101): {}, docValues:{}, maxPointsInLeafNode=386, maxMBSortInHeap=6.152622036236546, sim=Asserting(RandomSimilarity(queryNorm=false): {}), locale=sv-SE, timezone=America/Cayman
    NOTE: Linux 6.8.0-1021-azure amd64/Azul Systems, Inc. 21.0.6 (64-bit)/cpus=4,threads=1,free=383929800,total=536870912
    NOTE: All tests run in this JVM: [MLModelAutoReDeployerIT, RestBedRockInferenceIT, RestCohereInferenceIT]
  1> [2025-02-28T18:09:43,089][INFO ][o.o.m.r.RestCohereInferenceIT] [test_cohereInference_withDifferent_postProcessFunction] before test
  1> [2025-02-28T18:09:43,097][INFO ][o.o.m.r.RestCohereInferenceIT] [test_cohereInference_withDifferent_postProcessFunction] initializing REST clients against [http://[::1]:39439, http://127.0.0.1:45625]/
  1> [2025-02-28T18:09:44,756][INFO ][o.o.m.r.RestCohereInferenceIT] [test_cohereInference_withDifferent_postProcessFunction] after test

RestConnectorToolIT > testConnectorToolInFlowAgent STANDARD_OUT
    [2025-03-01T10:09:44,830][INFO ][o.o.m.r.RestConnectorToolIT] [testConnectorToolInFlowAgent] before test
    [2025-03-01T10:09:44,834][INFO ][o.o.m.r.RestConnectorToolIT] [testConnectorToolInFlowAgent] initializing REST clients against [http://[::1]:39439, http://127.0.0.1:45625]/

RestConnectorToolIT > testConnectorToolInFlowAgent STANDARD_ERROR
    માર્ચ 01, 2025 10:10:05 AM org.opensearch.client.RestClient logResponse
    WARNING: request [DELETE http://127.0.0.1:45625/.plugins-ml-agent] returned 1 warnings: [299 OpenSearch-3.0.0-SNAPSHOT-a961ec728859b5318a8c7f80206ff6566a954971 "this request accesses system indices: [.plugins-ml-agent], but in a future major version, direct access to system indices will be prevented by default"]

RestConnectorToolIT > testConnectorToolInFlowAgent STANDARD_OUT
    [2025-03-01T10:10:05,622][INFO ][o.o.m.r.RestConnectorToolIT] [testConnectorToolInFlowAgent] after test```

[dhrubo-os]

@mingshl I'm seeing to fail these tests in other PRs as well:

- org.opensearch.ml.rest.RestCohereInferenceIT.test_cohereInference_withDifferent_postProcessFunction
 - org.opensearch.ml.rest.RestMLInferenceSearchRequestProcessorIT.testMLInferenceProcessorRemoteModelOptionalInputs
 - org.opensearch.ml.rest.RestMLInferenceSearchResponseProcessorIT.testMLInferenceProcessorRemoteModelOptionalInputs

Could you please check? I feel like these aren't flaky tests. It's failing here as well: https://github.com/opensearch-project/ml-commons/actions/runs/13598168388/job/38019496540?pr=3597

[dhrubo-os]

Tests are failing here too: https://github.com/opensearch-project/ml-commons/actions/runs/13576646226/job/37971082585?pr=3595#logs

[mingshl]

@dhrubo-os this test is failing everywhere, but it's not related to search processors, it's the model interface related to cohere

116 tests completed, 1 failed, 11 skipped
Tests with failures:

• org.opensearch.ml.rest.RestCohereInferenceIT.test_cohereInference_withDifferent_postProcessFunction

[zane-neo]

@dhrubo-os this test is failing everywhere, but it's not related to search processors, it's the model interface related to cohere

116 tests completed, 1 failed, 11 skipped Tests with failures:

• org.opensearch.ml.rest.RestCohereInferenceIT.test_cohereInference_withDifferent_postProcessFunction

Cohere one is an issue and I fixed it in this PR: #3602

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.x 2.x
# Navigate to the new working tree
cd .worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-3598-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 4512e0ae34fd25fd1a5236c23aba48259f7fc2aa
# Push it to GitHub
git push --set-upstream origin backport/backport-3598-to-2.x
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-3598-to-2.x.