Skip to content

Commit 251bd44

Browse files
gaobinlonggaobinlong
committed
Merge remote-tracking branch 'upstream/main' into fix_match_phrase_prefix_bug
2 parents 8874093 + 3a3da4f commit 251bd44

File tree

49 files changed

+2372
-240
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

49 files changed

+2372
-240
lines changed

‎CHANGELOG.md

+6-3
Original file line numberDiff line numberDiff line change
@@ -162,25 +162,27 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
162162
### Changed
163163
- Mute the query profile IT with concurrent execution ([#9840](https://github.com/opensearch-project/OpenSearch/pull/9840))
164164
- Force merge with `only_expunge_deletes` honors max segment size ([#10036](https://github.com/opensearch-project/OpenSearch/pull/10036))
165-
- Add the means to extract the contextual properties from HttpChannel, TcpCChannel and TrasportChannel without excessive typecasting ([#10562](https://github.com/opensearch-project/OpenSearch/pull/10562))
165+
- Add the means to extract the contextual properties from HttpChannel, TcpCChannel and TrasportChannel without excessive typecasting ([#10562](https://github.com/opensearch-project/OpenSearch/pull/10562)), ([#11751](https://github.com/opensearch-project/OpenSearch/pull/11751))
166166
- Introduce new dynamic cluster setting to control slice computation for concurrent segment search ([#9107](https://github.com/opensearch-project/OpenSearch/pull/9107))
167167
- Search pipelines now support asynchronous request and response processors to avoid blocking on a transport thread ([#10598](https://github.com/opensearch-project/OpenSearch/pull/10598))
168168
- [Remote Store] Add Remote Store backpressure rejection stats to `_nodes/stats` ([#10524](https://github.com/opensearch-project/OpenSearch/pull/10524))
169169
- [BUG] Fix java.lang.SecurityException in repository-gcs plugin ([#10642](https://github.com/opensearch-project/OpenSearch/pull/10642))
170170
- Add telemetry tracer/metric enable flag and integ test. ([#10395](https://github.com/opensearch-project/OpenSearch/pull/10395))
171171
- Performance improvement for Datetime field caching ([#4558](https://github.com/opensearch-project/OpenSearch/issues/4558))
172172
- Add instrumentation for indexing in transport bulk action and transport shard bulk action. ([#10273](https://github.com/opensearch-project/OpenSearch/pull/10273))
173-
- Disallow removing some metadata fields by remove ingest processor ([#10895](https://github.com/opensearch-project/OpenSearch/pull/10895))
173+
- Disallow removing some metadata fields by remove ingest processor ([#10895](https://github.com/opensearch-project/OpenSearch/pull/10895), [#11607](https://github.com/opensearch-project/OpenSearch/pull/11607))
174174
- Performance improvement for MultiTerm Queries on Keyword fields ([#7057](https://github.com/opensearch-project/OpenSearch/issues/7057))
175175
- Refactor common parts from the Rounding class into a separate 'round' package ([#11023](https://github.com/opensearch-project/OpenSearch/issues/11023))
176176
- Performance improvement for date histogram aggregations without sub-aggregations ([#11083](https://github.com/opensearch-project/OpenSearch/pull/11083))
177177
- Disable concurrent aggs for Diversified Sampler and Sampler aggs ([#11087](https://github.com/opensearch-project/OpenSearch/issues/11087))
178178
- Made leader/follower check timeout setting dynamic ([#10528](https://github.com/opensearch-project/OpenSearch/pull/10528))
179+
- Improved performance of numeric exact-match queries ([#11209](https://github.com/opensearch-project/OpenSearch/pull/11209))
179180
- Change error message when per shard document limit is breached ([#11312](https://github.com/opensearch-project/OpenSearch/pull/11312))
180181
- Improve boolean parsing performance ([#11308](https://github.com/opensearch-project/OpenSearch/pull/11308))
181182
- Interpret byte array as primitive using VarHandles ([#11362](https://github.com/opensearch-project/OpenSearch/pull/11362))
182183
- Automatically add scheme to discovery.ec2.endpoint ([#11512](https://github.com/opensearch-project/OpenSearch/pull/11512))
183184
- Restore support for Java 8 for RestClient ([#11562](https://github.com/opensearch-project/OpenSearch/pull/11562))
185+
- Add deleted doc count in _cat/shards ([#11678](https://github.com/opensearch-project/OpenSearch/pull/11678))
184186

185187
### Deprecated
186188

@@ -189,7 +191,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
189191

190192
### Fixed
191193
- Fix failure in dissect ingest processor parsing empty brackets ([#9225](https://github.com/opensearch-project/OpenSearch/pull/9255))
192-
- Fix class_cast_exception when passing int to _version and other metadata fields in ingest simulate API ([#10101](https://github.com/opensearch-project/OpenSearch/pull/10101))
194+
- Fix `class_cast_exception` when passing int to `_version` and other metadata fields in ingest simulate API ([#10101](https://github.com/opensearch-project/OpenSearch/pull/10101))
193195
- Fix Segment Replication ShardLockObtainFailedException bug during index corruption ([10370](https://github.com/opensearch-project/OpenSearch/pull/10370))
194196
- Fix some test methods in SimulatePipelineRequestParsingTests never run and fix test failure ([#10496](https://github.com/opensearch-project/OpenSearch/pull/10496))
195197
- Fix match_phrase_prefix_query not working on text field with multiple values and index_prefixes ([#10959](https://github.com/opensearch-project/OpenSearch/pull/10959))
@@ -202,6 +204,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
202204
- Fix the issue with DefaultSpanScope restoring wrong span in the TracerContextStorage upon detach ([#11316](https://github.com/opensearch-project/OpenSearch/issues/11316))
203205
- Remove shadowJar from `lang-painless` module publication ([#11369](https://github.com/opensearch-project/OpenSearch/issues/11369))
204206
- Fix remote shards balancer and remove unused variables ([#11167](https://github.com/opensearch-project/OpenSearch/pull/11167))
207+
- Fix parsing of flat object fields with dots in keys ([#11425](https://github.com/opensearch-project/OpenSearch/pull/11425))
205208
- Fix bug where replication lag grows post primary relocation ([#11238](https://github.com/opensearch-project/OpenSearch/pull/11238))
206209
- Fix for stuck update action in a bulk with `retry_on_conflict` property ([#11152](https://github.com/opensearch-project/OpenSearch/issues/11152))
207210
- Fix template setting override for replication type ([#11417](https://github.com/opensearch-project/OpenSearch/pull/11417))

‎modules/analysis-common/src/yamlRestTest/resources/rest-api-spec/test/search.query/11_match_field_match_only_text.yml

+2-2
Original file line numberDiff line numberDiff line change
@@ -2,8 +2,8 @@
22

33
"match query with stacked stems":
44
- skip:
5-
version: " - 2.99.99"
6-
reason: "match_only_text was added in 3.0"
5+
version: " - 2.11.99"
6+
reason: "match_only_text was added in 2.12"
77
# Tests the match query stemmed tokens are "stacked" on top of the unstemmed
88
# versions in the same position.
99
- do:

‎modules/analysis-common/src/yamlRestTest/resources/rest-api-spec/test/search.query/20_ngram_search_field_match_only_text.yml

+4-4
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
"ngram search":
22
- skip:
3-
version: " - 2.99.99"
4-
reason: "match_only_text was added in 3.0"
3+
version: " - 2.11.99"
4+
reason: "match_only_text was added in 2.12"
55
- do:
66
indices.create:
77
index: test
@@ -45,8 +45,8 @@
4545
---
4646
"testNGramCopyField":
4747
- skip:
48-
version: " - 2.99.99"
49-
reason: "match_only_text was added in 3.0"
48+
version: " - 2.11.99"
49+
reason: "match_only_text was added in 2.12"
5050
- do:
5151
indices.create:
5252
index: test

‎modules/analysis-common/src/yamlRestTest/resources/rest-api-spec/test/search.query/30_ngram_highligthing_field_match_only_text.yml

+2-2
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
"ngram highlighting":
22
- skip:
3-
version: " - 2.99.99"
4-
reason: "match_only_text was added in 3.0"
3+
version: " - 2.11.99"
4+
reason: "match_only_text was added in 2.12"
55
- do:
66
indices.create:
77
index: test

‎modules/analysis-common/src/yamlRestTest/resources/rest-api-spec/test/search.query/40_query_string_field_match_only_text.yml

+2-2
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
11
---
22
"Test query string with snowball":
33
- skip:
4-
version: " - 2.99.99"
5-
reason: "match_only_text was added in 3.0"
4+
version: " - 2.11.99"
5+
reason: "match_only_text was added in 2.12"
66
- do:
77
indices.create:
88
index: test

‎modules/analysis-common/src/yamlRestTest/resources/rest-api-spec/test/search.query/41_query_string_with_default_analyzer_field_match_only_text.yml

+2-2
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
11
---
22
"Test default search analyzer is applied":
33
- skip:
4-
version: " - 2.99.99"
5-
reason: "match_only_text was added in 3.0"
4+
version: " - 2.11.99"
5+
reason: "match_only_text was added in 2.12"
66
- do:
77
indices.create:
88
index: test

‎modules/analysis-common/src/yamlRestTest/resources/rest-api-spec/test/search.query/50_queries_with_synonyms_field_match_only_text.yml

+4-4
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
11
---
22
"Test common terms query with stacked tokens":
33
- skip:
4-
version: " - 2.99.99"
5-
reason: "match_only_text was added in 3.0"
4+
version: " - 2.11.99"
5+
reason: "match_only_text was added in 2.12"
66
features: "allowed_warnings"
77

88
- do:
@@ -247,8 +247,8 @@
247247
---
248248
"Test match query with synonyms - see #3881 for extensive description of the issue":
249249
- skip:
250-
version: " - 2.99.99"
251-
reason: "match_only_text was added in 3.0"
250+
version: " - 2.11.99"
251+
reason: "match_only_text was added in 2.12"
252252
- do:
253253
indices.create:
254254
index: test

‎modules/analysis-common/src/yamlRestTest/resources/rest-api-spec/test/search.query/60_synonym_graph_field_match_only_text.yml

+2-2
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
setup:
22
- skip:
3-
version: " - 2.99.99"
4-
reason: "match_only_text was added in 3.0"
3+
version: " - 2.11.99"
4+
reason: "match_only_text was added in 2.12"
55
- do:
66
indices.create:
77
index: test

‎modules/analysis-common/src/yamlRestTest/resources/rest-api-spec/test/search.query/70_intervals_field_match_only_text.yml

+4-4
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
11
# integration tests for intervals queries using analyzers
22
setup:
33
- skip:
4-
version: " - 2.99.99"
5-
reason: "match_only_text was added in 3.0"
4+
version: " - 2.11.99"
5+
reason: "match_only_text was added in 2.12"
66
- do:
77
indices.create:
88
index: test
@@ -26,8 +26,8 @@ setup:
2626
---
2727
"Test use_field":
2828
- skip:
29-
version: " - 2.99.99"
30-
reason: "match_only_text was added in 3.0"
29+
version: " - 2.11.99"
30+
reason: "match_only_text was added in 2.12"
3131
- do:
3232
catch: bad_request
3333
search:

‎modules/analysis-common/src/yamlRestTest/resources/rest-api-spec/test/search.suggest/20_phrase_field_match_only_text.yml

+8-8
Original file line numberDiff line numberDiff line change
@@ -2,8 +2,8 @@
22

33
setup:
44
- skip:
5-
version: " - 2.99.99"
6-
reason: "match_only_text was added in 3.0"
5+
version: " - 2.11.99"
6+
reason: "match_only_text was added in 2.12"
77
- do:
88
indices.create:
99
index: test
@@ -122,8 +122,8 @@ setup:
122122
---
123123
"breaks ties by sorting terms":
124124
- skip:
125-
version: " - 2.99.99"
126-
reason: "match_only_text was added in 3.0"
125+
version: " - 2.11.99"
126+
reason: "match_only_text was added in 2.12"
127127
# This runs the suggester without bigrams so we can be sure of the sort order
128128
- do:
129129
search:
@@ -181,8 +181,8 @@ setup:
181181
---
182182
"doesn't fail when asked to run on a field without unigrams when force_unigrams=false":
183183
- skip:
184-
version: " - 2.99.99"
185-
reason: "match_only_text was added in 3.0"
184+
version: " - 2.11.99"
185+
reason: "match_only_text was added in 2.12"
186186
- do:
187187
search:
188188
rest_total_hits_as_int: true
@@ -213,8 +213,8 @@ setup:
213213
---
214214
"reverse suggestions":
215215
- skip:
216-
version: " - 2.99.99"
217-
reason: "match_only_text was added in 3.0"
216+
version: " - 2.11.99"
217+
reason: "match_only_text was added in 2.12"
218218
- do:
219219
search:
220220
rest_total_hits_as_int: true

‎modules/ingest-common/src/main/java/org/opensearch/ingest/common/RemoveProcessor.java

+11-9
Original file line numberDiff line numberDiff line change
@@ -88,16 +88,18 @@ public IngestDocument execute(IngestDocument document) {
8888
throw new IllegalArgumentException("cannot remove metadata field [" + path + "]");
8989
}
9090
// removing _id is disallowed when there's an external version specified in the request
91-
String versionType = document.getFieldValue(IngestDocument.Metadata.VERSION_TYPE.getFieldName(), String.class);
9291
if (path.equals(IngestDocument.Metadata.ID.getFieldName())
93-
&& !Objects.equals(versionType, VersionType.toString(VersionType.INTERNAL))) {
94-
Long version = document.getFieldValue(IngestDocument.Metadata.VERSION.getFieldName(), Long.class);
95-
throw new IllegalArgumentException(
96-
"cannot remove metadata field [_id] when specifying external version for the document, version: "
97-
+ version
98-
+ ", version_type: "
99-
+ versionType
100-
);
92+
&& document.hasField(IngestDocument.Metadata.VERSION_TYPE.getFieldName())) {
93+
String versionType = document.getFieldValue(IngestDocument.Metadata.VERSION_TYPE.getFieldName(), String.class);
94+
if (!Objects.equals(versionType, VersionType.toString(VersionType.INTERNAL))) {
95+
Long version = document.getFieldValue(IngestDocument.Metadata.VERSION.getFieldName(), Long.class, true);
96+
throw new IllegalArgumentException(
97+
"cannot remove metadata field [_id] when specifying external version for the document, version: "
98+
+ version
99+
+ ", version_type: "
100+
+ versionType
101+
);
102+
}
101103
}
102104
document.removeField(path);
103105
});

‎modules/ingest-common/src/test/java/org/opensearch/ingest/common/RemoveProcessorTests.java

+97
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,7 @@
3232

3333
package org.opensearch.ingest.common;
3434

35+
import org.opensearch.common.lucene.uid.Versions;
3536
import org.opensearch.index.VersionType;
3637
import org.opensearch.ingest.IngestDocument;
3738
import org.opensearch.ingest.Processor;
@@ -181,4 +182,100 @@ public void testRemoveMetadataField() throws Exception {
181182
}
182183
}
183184
}
185+
186+
public void testRemoveDocumentId() throws Exception {
187+
Map<String, Object> config = new HashMap<>();
188+
config.put("field", IngestDocument.Metadata.ID.getFieldName());
189+
String processorTag = randomAlphaOfLength(10);
190+
191+
// test remove _id when _version_type is external
192+
IngestDocument ingestDocumentWithExternalVersionType = new IngestDocument(
193+
RandomDocumentPicks.randomString(random()),
194+
RandomDocumentPicks.randomString(random()),
195+
RandomDocumentPicks.randomString(random()),
196+
1L,
197+
VersionType.EXTERNAL,
198+
RandomDocumentPicks.randomSource(random())
199+
);
200+
201+
Processor processorForExternalVersionType = new RemoveProcessor.Factory(TestTemplateService.instance()).create(
202+
null,
203+
processorTag,
204+
null,
205+
config
206+
);
207+
assertThrows(
208+
"cannot remove metadata field [_id] when specifying external version for the document, version: "
209+
+ 1
210+
+ ", version_type: "
211+
+ VersionType.EXTERNAL,
212+
IllegalArgumentException.class,
213+
() -> processorForExternalVersionType.execute(ingestDocumentWithExternalVersionType)
214+
);
215+
216+
// test remove _id when _version_type is external_gte
217+
config.put("field", IngestDocument.Metadata.ID.getFieldName());
218+
IngestDocument ingestDocumentWithExternalGTEVersionType = new IngestDocument(
219+
RandomDocumentPicks.randomString(random()),
220+
RandomDocumentPicks.randomString(random()),
221+
RandomDocumentPicks.randomString(random()),
222+
1L,
223+
VersionType.EXTERNAL_GTE,
224+
RandomDocumentPicks.randomSource(random())
225+
);
226+
227+
Processor processorForExternalGTEVersionType = new RemoveProcessor.Factory(TestTemplateService.instance()).create(
228+
null,
229+
processorTag,
230+
null,
231+
config
232+
);
233+
assertThrows(
234+
"cannot remove metadata field [_id] when specifying external version for the document, version: "
235+
+ 1
236+
+ ", version_type: "
237+
+ VersionType.EXTERNAL_GTE,
238+
IllegalArgumentException.class,
239+
() -> processorForExternalGTEVersionType.execute(ingestDocumentWithExternalGTEVersionType)
240+
);
241+
242+
// test remove _id when _version_type is internal
243+
config.put("field", IngestDocument.Metadata.ID.getFieldName());
244+
IngestDocument ingestDocumentWithInternalVersionType = new IngestDocument(
245+
RandomDocumentPicks.randomString(random()),
246+
RandomDocumentPicks.randomString(random()),
247+
RandomDocumentPicks.randomString(random()),
248+
Versions.MATCH_ANY,
249+
VersionType.INTERNAL,
250+
RandomDocumentPicks.randomSource(random())
251+
);
252+
253+
Processor processorForInternalVersionType = new RemoveProcessor.Factory(TestTemplateService.instance()).create(
254+
null,
255+
processorTag,
256+
null,
257+
config
258+
);
259+
processorForInternalVersionType.execute(ingestDocumentWithInternalVersionType);
260+
assertThat(ingestDocumentWithInternalVersionType.hasField(IngestDocument.Metadata.ID.getFieldName()), equalTo(false));
261+
262+
// test remove _id when _version_type is null
263+
config.put("field", IngestDocument.Metadata.ID.getFieldName());
264+
IngestDocument ingestDocumentWithNoVersionType = new IngestDocument(
265+
RandomDocumentPicks.randomString(random()),
266+
RandomDocumentPicks.randomString(random()),
267+
RandomDocumentPicks.randomString(random()),
268+
null,
269+
null,
270+
RandomDocumentPicks.randomSource(random())
271+
);
272+
Processor processorForNullVersionType = new RemoveProcessor.Factory(TestTemplateService.instance()).create(
273+
null,
274+
processorTag,
275+
null,
276+
config
277+
);
278+
processorForNullVersionType.execute(ingestDocumentWithNoVersionType);
279+
assertThat(ingestDocumentWithNoVersionType.hasField(IngestDocument.Metadata.ID.getFieldName()), equalTo(false));
280+
}
184281
}

0 commit comments

Comments
 (0)