Skip to content

Commit ee44250

Browse files
Merge branch 'main' into warm-index-recovery
2 parents d4d58dc + 47feca7 commit ee44250

File tree

211 files changed

+4949
-1679
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

211 files changed

+4949
-1679
lines changed

.github/CODEOWNERS

+2-2
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@
1111
# 3. Use the command palette to run the CODEOWNERS: Show owners of current file command, which will display all code owners for the current file.
1212

1313
# Default ownership for all repo files
14-
* @anasalkouz @andrross @Bukhtawar @CEHENKLE @dblock @dbwiddis @dreamer-89 @gbbafna @kotwanikunal @mch2 @msfroh @nknize @owaiskazi19 @reta @Rishikesh1159 @sachinpkale @saratvemulapalli @shwetathareja @sohami @tlfeng @VachaShah
14+
* @anasalkouz @andrross @Bukhtawar @CEHENKLE @dblock @dbwiddis @gbbafna @kotwanikunal @mch2 @msfroh @nknize @owaiskazi19 @reta @Rishikesh1159 @sachinpkale @saratvemulapalli @shwetathareja @sohami @VachaShah
1515

1616
/modules/transport-netty4/ @peternied
1717

@@ -24,4 +24,4 @@
2424

2525
/.github/ @peternied
2626

27-
/MAINTAINERS.md @anasalkouz @andrross @Bukhtawar @CEHENKLE @dblock @dbwiddis @dreamer-89 @gbbafna @jed326 @kotwanikunal @mch2 @msfroh @nknize @owaiskazi19 @peternied @reta @Rishikesh1159 @sachinpkale @saratvemulapalli @shwetathareja @sohami @tlfeng @VachaShah
27+
/MAINTAINERS.md @anasalkouz @andrross @Bukhtawar @CEHENKLE @dblock @dbwiddis @gbbafna @jed326 @kotwanikunal @mch2 @msfroh @nknize @owaiskazi19 @peternied @reta @Rishikesh1159 @sachinpkale @saratvemulapalli @shwetathareja @sohami @VachaShah

CHANGELOG.md

+21
Original file line numberDiff line numberDiff line change
@@ -6,21 +6,42 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
66
## [Unreleased 2.x]
77
### Added
88
- Add fingerprint ingest processor ([#13724](https://github.com/opensearch-project/OpenSearch/pull/13724))
9+
- [Remote Store] Rate limiter for remote store low priority uploads ([#14374](https://github.com/opensearch-project/OpenSearch/pull/14374/))
10+
- Apply the date histogram rewrite optimization to range aggregation ([#13865](https://github.com/opensearch-project/OpenSearch/pull/13865))
911
- [Writable Warm] Add composite directory implementation and integrate it with FileCache ([12782](https://github.com/opensearch-project/OpenSearch/pull/12782))
1012

1113
### Dependencies
1214
- Bump `org.gradle.test-retry` from 1.5.8 to 1.5.9 ([#13442](https://github.com/opensearch-project/OpenSearch/pull/13442))
1315
- Update to Apache Lucene 9.11.0 ([#14042](https://github.com/opensearch-project/OpenSearch/pull/14042))
1416
- Bump `netty` from 4.1.110.Final to 4.1.111.Final ([#14356](https://github.com/opensearch-project/OpenSearch/pull/14356))
1517
- Bump `org.wiremock:wiremock-standalone` from 3.3.1 to 3.6.0 ([#14361](https://github.com/opensearch-project/OpenSearch/pull/14361))
18+
- Bump `reactor` from 3.5.17 to 3.5.18 ([#14395](https://github.com/opensearch-project/OpenSearch/pull/14395))
19+
- Bump `reactor-netty` from 1.1.19 to 1.1.20 ([#14395](https://github.com/opensearch-project/OpenSearch/pull/14395))
20+
- Bump `commons-net:commons-net` from 3.10.0 to 3.11.1 ([#14396](https://github.com/opensearch-project/OpenSearch/pull/14396))
21+
- Bump `com.nimbusds:nimbus-jose-jwt` from 9.37.3 to 9.40 ([#14398](https://github.com/opensearch-project/OpenSearch/pull/14398))
22+
- Bump `org.apache.commons:commons-configuration2` from 2.10.1 to 2.11.0 ([#14399](https://github.com/opensearch-project/OpenSearch/pull/14399))
23+
- Bump `com.gradle.develocity` from 3.17.4 to 3.17.5 ([#14397](https://github.com/opensearch-project/OpenSearch/pull/14397))
24+
- Bump `opentelemetry` from 1.36.0 to 1.39.0 ([#14457](https://github.com/opensearch-project/OpenSearch/pull/14457))
25+
- Bump `azure-identity` from 1.11.4 to 1.13.0, Bump `msal4j` from 1.14.3 to 1.15.1, Bump `msal4j-persistence-extension` from 1.2.0 to 1.3.0 ([#14506](https://github.com/opensearch-project/OpenSearch/pull/14506))
1626

1727
### Changed
28+
- unsignedLongRangeQuery now returns MatchNoDocsQuery if the lower bounds are greater than the upper bounds ([#14416](https://github.com/opensearch-project/OpenSearch/pull/14416))
29+
- Updated the `indices.query.bool.max_clause_count` setting from being static to dynamically updateable ([#13568](https://github.com/opensearch-project/OpenSearch/pull/13568))
30+
- Make the class CommunityIdProcessor final ([#14448](https://github.com/opensearch-project/OpenSearch/pull/14448))
1831

1932
### Deprecated
2033

2134
### Removed
2235

2336
### Fixed
37+
- Fix bug in SBP cancellation logic ([#13259](https://github.com/opensearch-project/OpenSearch/pull/13474))
38+
- Fix handling of Short and Byte data types in ScriptProcessor ingest pipeline ([#14379](https://github.com/opensearch-project/OpenSearch/issues/14379))
39+
- Switch to iterative version of WKT format parser ([#14086](https://github.com/opensearch-project/OpenSearch/pull/14086))
40+
- Fix the computed max shards of cluster to avoid int overflow ([#14155](https://github.com/opensearch-project/OpenSearch/pull/14155))
41+
- Fixed rest-high-level client searchTemplate & mtermVectors endpoints to have a leading slash ([#14465](https://github.com/opensearch-project/OpenSearch/pull/14465))
42+
- Write shard level metadata blob when snapshotting searchable snapshot indexes ([#13190](https://github.com/opensearch-project/OpenSearch/pull/13190))
43+
- Fix aggs result of NestedAggregator with sub NestedAggregator ([#13324](https://github.com/opensearch-project/OpenSearch/pull/13324))
44+
- Add ListPitInfo::getKeepAlive() getter ([#14495](https://github.com/opensearch-project/OpenSearch/pull/14495))
2445

2546
### Security
2647

DEVELOPER_GUIDE.md

+10-8
Original file line numberDiff line numberDiff line change
@@ -651,16 +651,18 @@ Note that these snapshots do not follow the Maven [naming convention](https://ma
651651

652652
### Flaky Tests
653653

654-
OpenSearch has a very large test suite with long running, often failing (flaky), integration tests. Such individual tests are labelled as [Flaky Random Test Failure](https://github.com/opensearch-project/OpenSearch/issues?q=is%3Aopen+is%3Aissue+label%3A%22flaky-test%22). Your help is wanted fixing these!
654+
If you encounter a test failure locally or in CI that is seemingly unrelated to the change in your pull request, it may be a known flaky test or a new test failure. OpenSearch has a very large test suite with long running, often failing (flaky), integration tests. Such individual tests are labelled as [Flaky Random Test Failure](https://github.com/opensearch-project/OpenSearch/issues?q=is%3Aopen+is%3Aissue+label%3A%22flaky-test%22). Your help is wanted fixing these!
655655

656-
If you encounter a build/test failure in CI that is unrelated to the change in your pull request, it may be a known flaky test, or a new test failure.
656+
The automation [gradle-check-flaky-test-detector](https://build.ci.opensearch.org/job/gradle-check-flaky-test-detector/), which runs in OpenSearch public Jenkins, identifies failing flaky issues that are part of post-merge actions. Once a flaky test is identified, the automation creates an issue with detailed report that includes links to all relevant commits, the Gradle check build log, the test report, and pull requests that are impacted with the flaky test failures. This automation leverages data from the [OpenSearch Metrics Project](https://github.com/opensearch-project/opensearch-metrics) to establish a baseline for creating the issue and updating the flaky test report. For all flaky test issues created by automation, visit this [link](https://github.com/opensearch-project/OpenSearch/issues?q=is%3Aissue+is%3Aopen+label%3A%3Etest-failure+author%3Aopensearch-ci-bot).
657+
658+
If you still see a failing test that is not part of the post merge actions, please do:
659+
660+
* Follow failed CI links, and locate the failing test(s) or use the [Gradle Check Metrics Dashboard](#gradle-check-metrics-dashboard).
661+
* Copy-paste the failure into a comment of your PR.
662+
* Search through issues using the name of the failed test for whether this is a known flaky test.
663+
* If no existing issue is found, open one.
664+
* Retry CI via the GitHub UX or by pushing an update to your PR.
657665

658-
1. Follow failed CI links, and locate the failing test(s).
659-
2. Copy-paste the failure into a comment of your PR.
660-
3. Search through [issues](https://github.com/opensearch-project/OpenSearch/issues?q=is%3Aopen+is%3Aissue+label%3A%22flaky-test%22) using the name of the failed test for whether this is a known flaky test.
661-
4. If an existing issue is found, paste a link to the known issue in a comment to your PR.
662-
5. If no existing issue is found, open one.
663-
6. Retry CI via the GitHub UX or by pushing an update to your PR.
664666

665667
### Gradle Check Metrics Dashboard
666668

MAINTAINERS.md

+11-11
Original file line numberDiff line numberDiff line change
@@ -26,18 +26,18 @@ This document contains a list of maintainers in this repo. See [opensearch-proje
2626
| Sarat Vemulapalli | [saratvemulapalli](https://github.com/saratvemulapalli) | Amazon |
2727
| Shweta Thareja | [shwetathareja](https://github.com/shwetathareja) | Amazon |
2828
| Sorabh Hamirwasia | [sohami](https://github.com/sohami) | Amazon |
29-
| Suraj Singh | [dreamer-89](https://github.com/dreamer-89) | Amazon |
30-
| Tianli Feng | [tlfeng](https://github.com/tlfeng) | Amazon |
3129
| Vacha Shah | [VachaShah](https://github.com/VachaShah) | Amazon |
3230

3331
## Emeritus
3432

35-
| Maintainer | GitHub ID | Affiliation |
36-
| --------------------- | ----------------------------------------- | ----------- |
37-
| Megha Sai Kavikondala | [meghasaik](https://github.com/meghasaik) | Amazon |
38-
| Xue Zhou | [xuezhou25](https://github.com/xuezhou25) | Amazon |
39-
| Kartik Ganesh | [kartg](https://github.com/kartg) | Amazon |
40-
| Abbas Hussain | [abbashus](https://github.com/abbashus) | Meta |
41-
| Himanshu Setia | [setiah](https://github.com/setiah) | Amazon |
42-
| Ryan Bogan | [ryanbogan](https://github.com/ryanbogan) | Amazon |
43-
| Rabi Panda | [adnapibar](https://github.com/adnapibar) | Independent |
33+
| Maintainer | GitHub ID | Affiliation |
34+
| ---------------------- |-------------------------------------------- | ----------- |
35+
| Megha Sai Kavikondala | [meghasaik](https://github.com/meghasaik) | Amazon |
36+
| Xue Zhou | [xuezhou25](https://github.com/xuezhou25) | Amazon |
37+
| Kartik Ganesh | [kartg](https://github.com/kartg) | Amazon |
38+
| Abbas Hussain | [abbashus](https://github.com/abbashus) | Meta |
39+
| Himanshu Setia | [setiah](https://github.com/setiah) | Amazon |
40+
| Ryan Bogan | [ryanbogan](https://github.com/ryanbogan) | Amazon |
41+
| Rabi Panda | [adnapibar](https://github.com/adnapibar) | Independent |
42+
| Tianli Feng | [tlfeng](https://github.com/tlfeng) | Amazon |
43+
| Suraj Singh | [dreamer-89](https://github.com/dreamer-89) | Amazon |

TESTING.md

+33-7
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,9 @@ OpenSearch uses [jUnit](https://junit.org/junit5/) for testing, it also uses ran
3333
- [Bad practices](#bad-practices)
3434
- [Use randomized-testing for coverage](#use-randomized-testing-for-coverage)
3535
- [Abuse randomization in multi-threaded tests](#abuse-randomization-in-multi-threaded-tests)
36+
- [Use `Thread.sleep`](#use-threadsleep)
37+
- [Expect a specific segment topology](#expect-a-specific-segment-topology)
38+
- [Leave environment in an unstable state after test](#leave-environment-in-an-unstable-state-after-test)
3639
- [Test coverage analysis](#test-coverage-analysis)
3740
- [Building with extra plugins](#building-with-extra-plugins)
3841
- [Environment misc](#environment-misc)
@@ -88,21 +91,23 @@ This will instruct all JVMs (including any that run cli tools such as creating t
8891

8992
## Test case filtering
9093

91-
- `tests.class` is a class-filtering shell-like glob pattern
92-
- `tests.method` is a method-filtering glob pattern.
94+
To be able to run a single test you need to specify the module where you're running the tests from.
95+
96+
Example: `./gradlew server:test --tests "*.ReplicaShardBatchAllocatorTests.testNoAsyncFetchData"`
9397

9498
Run a single test case (variants)
9599

96-
./gradlew test -Dtests.class=org.opensearch.package.ClassName
97-
./gradlew test "-Dtests.class=*.ClassName"
100+
./gradlew module:test --tests org.opensearch.package.ClassName
101+
./gradlew module:test --tests org.opensearch.package.ClassName.testName
102+
./gradlew module:test --tests "*.ClassName"
98103

99104
Run all tests in a package and its sub-packages
100105

101-
./gradlew test "-Dtests.class=org.opensearch.package.*"
106+
./gradlew module:test --tests "org.opensearch.package.*"
102107

103108
Run any test methods that contain *esi* (e.g.: .r*esi*ze.)
104109

105-
./gradlew test "-Dtests.method=*esi*"
110+
./gradlew module:test --tests "*esi*"
106111

107112
Run all tests that are waiting for a bugfix (disabled by default)
108113

@@ -455,7 +460,7 @@ Unit tests are the preferred way to test some functionality: most of the time th
455460

456461
The reason why `OpenSearchSingleNodeTestCase` exists is that all our components used to be very hard to set up in isolation, which had led us to having a number of integration tests but close to no unit tests. `OpenSearchSingleNodeTestCase` is a workaround for this issue which provides an easy way to spin up a node and get access to components that are hard to instantiate like `IndicesService`. Whenever practical, you should prefer unit tests.
457462

458-
Finally, if the the functionality under test needs to be run in a cluster, there are two test classes to consider:
463+
Finally, if the functionality under test needs to be run in a cluster, there are two test classes to consider:
459464
* `OpenSearchRestTestCase` will connect to an external cluster. This is a good option if the tests cases don't rely on a specific configuration of the test cluster. A test cluster is set up as part of the Gradle task running integration tests, and test cases using this class can connect to it. The configuration of the cluster is provided in the Gradle files.
460465
* `OpenSearchIntegTestCase` will create a local cluster as part of each test case. The configuration of the cluster is controlled by the test class. This is a good option if different tests cases depend on different cluster configurations, as it would be impractical (and limit parallelization) to keep re-configuring (and re-starting) the external cluster for each test case. A good example of when this class might come in handy is for testing security features, where different cluster configurations are needed to fully test each one.
461466

@@ -477,6 +482,27 @@ However, it should not be used for coverage. For instance if you are testing a p
477482

478483
Multi-threaded tests are often not reproducible due to the fact that there is no guarantee on the order in which operations occur across threads. Adding randomization to the mix usually makes things worse and should be done with care.
479484

485+
### Use `Thread.sleep`
486+
487+
`Thread.sleep()` is almost always a bad idea because it is very difficult to know that you've waited long enough. Using primitives like `waitUntil` or `assertBusy`, which use Thread.sleep internally, is okay to wait for a specific condition. However, it is almost always better to instrument your code with concurrency primitives like a `CountDownLatch` that will allow you to deterministically wait for a specific condition, without waiting longer than necessary that will happen with a polling approach used by `assertBusy`.
488+
489+
Example:
490+
- [PrimaryShardAllocatorIT](https://github.com/opensearch-project/OpenSearch/blob/7ffcd6500e0bd5956cef5c289ee66d9f99d533fc/server/src/internalClusterTest/java/org/opensearch/gateway/ReplicaShardAllocatorIT.java#L208-L235): This test is using two latches: one to wait for a recovery to start and one to block that recovery so that it can deterministically test things that happen during a recovery.
491+
492+
### Expect a specific segment topology
493+
494+
By design, OpenSearch integration tests will vary how the merge policy works because in almost all scenarios you should not depend on a specific segment topology (in the real world your code will see a huge diversity of indexing workloads with OpenSearch merging things in the background all the time!). If you do in fact need to care about the segment topology (e.g. for testing statistics that might vary slightly depending on number of segments), then you must take care to ensure that segment topology is deterministic by doing things like disabling background refreshes, force merging after indexing data, etc.
495+
496+
Example:
497+
- [SegmentReplicationResizeRequestIT](https://github.com/opensearch-project/OpenSearch/blob/f715ee1a485e550802accc1c2e3d8101208d4f0b/server/src/internalClusterTest/java/org/opensearch/indices/replication/SegmentReplicationResizeRequestIT.java#L102-L109): This test disables refreshes to prevent interfering with the segment replication behavior under test.
498+
499+
### Leave environment in an unstable state after test
500+
501+
The default test case will ensure that no open file handles or running threads are left after tear down. You must ensure that all resources are cleaned up at the end of each test case, or else the cleanup may end up racing with the tear down logic in the base test class in a way that is very difficult to reproduce.
502+
503+
Example:
504+
- [AwarenessAttributeDecommissionIT](https://github.com/opensearch-project/OpenSearch/blob/main/server/src/internalClusterTest/java/org/opensearch/cluster/coordination/AwarenessAttributeDecommissionIT.java#L951): Recommissions any decommissioned nodes at the end of the test to ensure the after-test checks succeed.
505+
480506
# Test coverage analysis
481507

482508
The code coverage report can be generated through Gradle with [JaCoCo plugin](https://docs.gradle.org/current/userguide/jacoco_plugin.html).

buildSrc/src/main/java/org/opensearch/gradle/testfixtures/TestFixturesPlugin.java

+2
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,7 @@
3434
import com.avast.gradle.dockercompose.ComposeExtension;
3535
import com.avast.gradle.dockercompose.DockerComposePlugin;
3636
import com.avast.gradle.dockercompose.ServiceInfo;
37+
import com.avast.gradle.dockercompose.tasks.ComposeBuild;
3738
import com.avast.gradle.dockercompose.tasks.ComposeDown;
3839
import com.avast.gradle.dockercompose.tasks.ComposePull;
3940
import com.avast.gradle.dockercompose.tasks.ComposeUp;
@@ -200,6 +201,7 @@ public void execute(Task task) {
200201
maybeSkipTasks(tasks, dockerSupport, getTaskClass("org.opensearch.gradle.test.RestIntegTestTask"));
201202
maybeSkipTasks(tasks, dockerSupport, TestingConventionsTasks.class);
202203
maybeSkipTasks(tasks, dockerSupport, getTaskClass("org.opensearch.gradle.test.AntFixture"));
204+
maybeSkipTasks(tasks, dockerSupport, ComposeBuild.class);
203205
maybeSkipTasks(tasks, dockerSupport, ComposeUp.class);
204206
maybeSkipTasks(tasks, dockerSupport, ComposePull.class);
205207
maybeSkipTasks(tasks, dockerSupport, ComposeDown.class);

buildSrc/version.properties

+5-5
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
opensearch = 3.0.0
2-
lucene = 9.11.0
2+
lucene = 9.12.0-snapshot-c896995
33

44
bundled_jdk_vendor = adoptium
55
bundled_jdk = 21.0.3+9
@@ -33,8 +33,8 @@ netty = 4.1.111.Final
3333
joda = 2.12.7
3434

3535
# project reactor
36-
reactor_netty = 1.1.19
37-
reactor = 3.5.17
36+
reactor_netty = 1.1.20
37+
reactor = 3.5.18
3838

3939
# client dependencies
4040
httpclient5 = 5.2.1
@@ -74,5 +74,5 @@ jzlib = 1.1.3
7474
resteasy = 6.2.4.Final
7575

7676
# opentelemetry dependencies
77-
opentelemetry = 1.36.0
78-
opentelemetrysemconv = 1.23.1-alpha
77+
opentelemetry = 1.39.0
78+
opentelemetrysemconv = 1.25.0-alpha

client/rest-high-level/src/main/java/org/opensearch/client/RequestConverters.java

+2-3
Original file line numberDiff line numberDiff line change
@@ -540,7 +540,7 @@ static Request searchTemplate(SearchTemplateRequest searchTemplateRequest) throw
540540
Request request;
541541

542542
if (searchTemplateRequest.isSimulate()) {
543-
request = new Request(HttpGet.METHOD_NAME, "_render/template");
543+
request = new Request(HttpGet.METHOD_NAME, "/_render/template");
544544
} else {
545545
SearchRequest searchRequest = searchTemplateRequest.getRequest();
546546
String endpoint = endpoint(searchRequest.indices(), "_search/template");
@@ -803,8 +803,7 @@ static Request termVectors(TermVectorsRequest tvrequest) throws IOException {
803803
}
804804

805805
static Request mtermVectors(MultiTermVectorsRequest mtvrequest) throws IOException {
806-
String endpoint = "_mtermvectors";
807-
Request request = new Request(HttpGet.METHOD_NAME, endpoint);
806+
Request request = new Request(HttpGet.METHOD_NAME, "/_mtermvectors");
808807
request.setEntity(createEntity(mtvrequest, REQUEST_BODY_CONTENT_TYPE));
809808
return request;
810809
}

0 commit comments

Comments
 (0)