Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Backport 2.x] fix optional mappings in ml inference search processors #3595

Merged
merged 2 commits into from
Mar 2, 2025
Merged

[Backport 2.x] fix optional mappings in ml inference search processors #3595

merged 2 commits into from
Mar 2, 2025

Conversation

opensearch-trigger-bot[bot]
Copy link
Contributor

Backport b22e61a from #3587

@mingshl @github-actions
fix optional mappings in ml inference search processors (#3587)
da76adb 
* fix optional mappings

Signed-off-by: Mingshi Liu <mingshl@amazon.com>

* use collections and add more assertion tests

Signed-off-by: Mingshi Liu <mingshl@amazon.com>

* validate query return false

Signed-off-by: Mingshi Liu <mingshl@amazon.com>

---------

Signed-off-by: Mingshi Liu <mingshl@amazon.com>
(cherry picked from commit b22e61a)
@mingshl
fix flaky test (#3598)
ab0b3ec 
Signed-off-by: Mingshi Liu <mingshl@amazon.com>
@mingshl
Copy link
Collaborator

mingshl commented Mar 1, 2025

backport #3598 together

@mingshl mingshl had a problem deploying to ml-commons-cicd-env March 1, 2025 06:06 — with GitHub Actions Error
@mingshl mingshl had a problem deploying to ml-commons-cicd-env March 1, 2025 06:06 — with GitHub Actions Failure
@mingshl mingshl had a problem deploying to ml-commons-cicd-env March 1, 2025 06:06 — with GitHub Actions Error
@mingshl mingshl had a problem deploying to ml-commons-cicd-env March 1, 2025 06:06 — with GitHub Actions Error
@mingshl mingshl had a problem deploying to ml-commons-cicd-env March 1, 2025 06:06 — with GitHub Actions Failure
@mingshl mingshl had a problem deploying to ml-commons-cicd-env March 1, 2025 06:06 — with GitHub Actions Error
@mingshl mingshl temporarily deployed to ml-commons-cicd-env March 1, 2025 18:13 — with GitHub Actions Inactive
@mingshl mingshl had a problem deploying to ml-commons-cicd-env March 1, 2025 18:13 — with GitHub Actions Error
@mingshl mingshl temporarily deployed to ml-commons-cicd-env March 1, 2025 18:13 — with GitHub Actions Inactive
@mingshl mingshl had a problem deploying to ml-commons-cicd-env March 1, 2025 18:13 — with GitHub Actions Error
@mingshl mingshl temporarily deployed to ml-commons-cicd-env March 1, 2025 18:13 — with GitHub Actions Inactive
@mingshl mingshl had a problem deploying to ml-commons-cicd-env March 1, 2025 18:13 — with GitHub Actions Failure
@mingshl
Copy link
Collaborator

mingshl commented Mar 1, 2025

flaky tests not related

2> REPRODUCE WITH: gradlew ':opensearch-ml-plugin:test' --tests "org.opensearch.ml.action.prediction.PredictionITTests.testPredictionWithSearchInput_LogisticRegression" -Dtests.seed=FD512049A5D88E17 -Dtests.security.manager=false -Dtests.locale=ko -Dtests.timezone=Asia/Yekaterinburg -Druntime.java=11  2> 
  2> java.util.ConcurrentModificationException
        at __randomizedtesting.SeedInfo.seed([FD512049A5D88E17:D64C731720B7CFFD]:0)
        at java.base/java.util.HashMap$ValueSpliterator.forEachRemaining(HashMap.java:1698)
        at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658)
        at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:274)
        at java.base/java.util.WeakHashMap$ValueSpliterator.forEachRemaining(WeakHashMap.java:1217)
        at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
        at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
        at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
        at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
        at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
        at org.apache.logging.log4j.core.LoggerContext.updateLoggers(LoggerContext.java:776)
        at org.apache.logging.log4j.core.LoggerContext.updateLoggers(LoggerContext.java:766)
        at org.opensearch.common.logging.Loggers.removeAppender(Loggers.java:176)
        at org.opensearch.test.OpenSearchTestCase.removeHeaderWarningAppender(OpenSearchTestCase.java:411)
        at jdk.internal.reflect.GeneratedMethodAccessor41.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
        at org.opensearch.test.OpenSearchTestClusterRule$1.evaluate(OpenSearchTestClusterRule.java:369)
        at org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:258)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
        at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
        at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
        at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
        at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
        at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
        at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:829)
  2> 3? 01, 2025 11:37:43 ?? org.tribuo.common.sgd.AbstractSGDTrainer train
  2> ??: Training SGD model with 100 examples
  2> 3? 01, 2025 11:37:43 ?? org.tribuo.common.sgd.AbstractSGDTrainer train
  2> ??: Outputs - RegressionInfo({name=price,id=0,count=100,max=99.000000,min=0.000000,mean=49.500000,variance=841.666667})
  2> 3? 01, 2025 11:37:43 ?? org.tribuo.common.sgd.AbstractSGDTrainer train
  2> ??: Training SGD model with 150 examples
  2> 3? 01, 2025 11:37:43 ?? org.tribuo.common.sgd.AbstractSGDTrainer train
  2> ??: Outputs - (0,Iris-versicolor,50), (1,Iris-virginica,50), (2,Iris-setosa,50)
  2> 3? 01, 2025 11:37:45 ?? org.tribuo.common.sgd.AbstractSGDTrainer train
  2> ??: Training SGD model with 100 examples
  2> 3? 01, 2025 11:37:45 ?? org.tribuo.common.sgd.AbstractSGDTrainer train
  2> ??: Outputs - RegressionInfo({name=price,id=0,count=100,max=99.000000,min=0.000000,mean=49.500000,variance=841.666667})
  2> 3? 01, 2025 11:37:45 ?? org.tribuo.common.sgd.AbstractSGDTrainer train
  2> ??: Training SGD model with 150 examples
  2> 3? 01, 2025 11:37:45 ?? org.tribuo.common.sgd.AbstractSGDTrainer train
  2> ??: Outputs - (0,Iris-versicolor,50), (1,Iris-virginica,50), (2,Iris-setosa,50)
  2> 3? 01, 2025 11:37:48 ?? org.tribuo.common.sgd.AbstractSGDTrainer train
  2> ??: Training SGD model with 100 examples
  2> 3? 01, 2025 11:37:48 ?? org.tribuo.common.sgd.AbstractSGDTrainer train
  2> ??: Outputs - RegressionInfo({name=price,id=0,count=100,max=99.000000,min=0.000000,mean=49.500000,variance=841.666667})
  2> 3? 01, 2025 11:37:48 ?? org.tribuo.common.sgd.AbstractSGDTrainer train
  2> ??: Training SGD model with 150 examples
  2> 3? 01, 2025 11:37:48 ?? org.tribuo.common.sgd.AbstractSGDTrainer train
  2> ??: Outputs - (0,Iris-versicolor,50), (1,Iris-virginica,50), (2,Iris-setosa,50)
  2> NOTE: leaving temporary files on disk at: D:\a\ml-commons\ml-commons\plugin\build\testrun\test\temp\org.opensearch.ml.action.prediction.PredictionITTests_FD512049A5D88E17-001  2> 
  2> NOTE: test params are: codec=Asserting(Lucene912): {index_uuid=PostingsFormat(name=Lucene912), class.keyword=PostingsFormat(name=Asserting), _id=PostingsFormat(name=Asserting), type=PostingsFormat(name=Lucene912), class=PostingsFormat(name=Lucene912)}, docValues:{_seq_no=DocValuesFormat(name=Asserting), sepal_width_in_cm=DocValuesFormat(name=Asserting), class.keyword=DocValuesFormat(name=Asserting), sepal_length_in_cm=DocValuesFormat(name=Asserting), _primary_term=DocValuesFormat(name=Asserting), _version=DocValuesFormat(name=Asserting), petal_width_in_cm=DocValuesFormat(name=Asserting), petal_length_in_cm=DocValuesFormat(name=Asserting)}, maxPointsInLeafNode=473, maxMBSortInHeap=6.91958077518492, sim=Asserting(RandomSimilarity(queryNorm=true): {}), locale=ko, timezone=Asia/Yekaterinburg  2> 
  2> NOTE: Windows Server 2022 10.0 amd64/Azul Systems, Inc. 11.0.26 (64-bit)/cpus=4,threads=1,free=4[2939](https://github.com/opensearch-project/ml-commons/actions/runs/13601827847/job/38028669488?pr=3595#step:5:2940)2640,total=536870912  2> 
  2> NOTE: All tests run in this JVM: [RegisterAgentTransportActionTests, TransportBatchIngestionActionTests, DeleteConnectorTransportActionTests, GetConnectorTransportActionTests, TransportCreateConnectorActionTests, CreateControllerTransportActionTests, CustomModelITTests, TransportDeployModelActionTests, ExecuteITTests, MLSearchHandlerTests, GetModelGroupITTests, RegisterModelGroupITTests, SearchModelGroupTransportActionTests, TransportUpdateModelGroupActionTests, DeleteModelTransportActionTests, GetModelTransportActionTests, SearchModelTransportActionTests, PredictionITTests]  2> 
  1> [2025-03-01T11:37:30,066][INFO ][o.o.p.PluginsService     ] [testPredictionWithoutDataset_KMeans] loaded plugin [org.opensearch.test.transport.MockTransportService$TestPlugin]
  1> [2025-03-01T11:37:30,066][INFO ][o.o.p.PluginsService     ] [testPredictionWithoutDataset_KMeans] loaded plugin [org.opensearch.transport.nio.MockNioTransportPlugin]
  1> [2025-03-01T11:37:30,066][INFO ][o.o.e.NodeEnvironment    ] [testPredictionWithoutDataset_KMeans] using [1] data paths, mounts [[Temporary Storage (D:)]], net usable_space [143.2gb], net total_space [149.9gb], types [NTFS]
  1> [2025-03-01T11:37:30,066][INFO ][o.o.e.NodeEnvironment    ] [testPredictionWithoutDataset_KMeans] heap size [512mb], compressed ordinary object pointers [true]
  1> [2025-03-01T11:37:30,066][INFO ][o.o.n.Node               ] [testPredictionWithoutDataset_KMeans] node name [node_s2], node ID [hXjkaWwESE2oa_UbtYNOiQ], cluster name [SUITE-TEST_WORKER_VM=[6]-CLUSTER_SEED=[-3260172273171154498]-HASH=[22C8279D474]-cluster], roles [ingest, remote_cluster_client, data]

@mingshl mingshl merged commit b09b8f6 into 2.x Mar 2, 2025
10 of 14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dhrubo-os dhrubo-os dhrubo-os approved these changes

@mingshl mingshl mingshl approved these changes

@b4sjoo b4sjoo Awaiting requested review from b4sjoo b4sjoo is a code owner

@jngz-es jngz-es Awaiting requested review from jngz-es jngz-es is a code owner

@model-collapse model-collapse Awaiting requested review from model-collapse model-collapse is a code owner

@rbhavna rbhavna Awaiting requested review from rbhavna rbhavna is a code owner

@ylwu-amzn ylwu-amzn Awaiting requested review from ylwu-amzn ylwu-amzn is a code owner

@zane-neo zane-neo Awaiting requested review from zane-neo zane-neo is a code owner

@Zhangxunmt Zhangxunmt Awaiting requested review from Zhangxunmt Zhangxunmt is a code owner

@austintlee austintlee Awaiting requested review from austintlee austintlee is a code owner

@HenryL27 HenryL27 Awaiting requested review from HenryL27 HenryL27 is a code owner

@xinyual xinyual Awaiting requested review from xinyual xinyual is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@mingshl @dhrubo-os