Skip to content

Commit 2ac6e68

Browse files
committed
Merge remote-tracking branch 'origin/main' into views-core
Signed-off-by: Peter Nied <peternied@hotmail.com>
2 parents bbf4945 + 95a1819 commit 2ac6e68

File tree

42 files changed

+521
-76
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

42 files changed

+521
-76
lines changed

.github/ISSUE_TEMPLATE/meta.yml

+33-3
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,36 @@ body:
2323
placeholder: Ex. - [ ] https://github.com/opensearch-project/security/issues/3888 Add views to the cluster metadata schema
2424
validations:
2525
required: true
26-
27-
28-
26+
- type: dropdown
27+
attributes:
28+
label: Related component
29+
description: Choose a specific OpenSearch component your project belongs to. If you are unsure of which component to select or if the component is not present, select "Other".
30+
multiple: false
31+
options:
32+
- # Empty first option to force selection
33+
- Build
34+
- Clients
35+
- Cluster Manager
36+
- Extensions
37+
- Indexing:Performance
38+
- Indexing:Replication
39+
- Indexing
40+
- Libraries
41+
- Other
42+
- Plugins
43+
- Search:Aggregations
44+
- Search:Performance
45+
- Search:Query Capabilities
46+
- Search:Query Insights
47+
- Search:Relevance
48+
- Search:Remote Search
49+
- Search:Resiliency
50+
- Search:Searchable Snapshots
51+
- Search
52+
- Storage:Durability
53+
- Storage:Performance
54+
- Storage:Remote
55+
- Storage:Snapshots
56+
- Storage
57+
validations:
58+
required: true

CHANGELOG.md

+9-1
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
1717
- [Admission Control] Integrate CPU AC with ResourceUsageCollector and add CPU AC stats to nodes/stats ([#10887](https://github.com/opensearch-project/OpenSearch/pull/10887))
1818
- [S3 Repository] Add setting to control connection count for sync client ([#12028](https://github.com/opensearch-project/OpenSearch/pull/12028))
1919
- Views, simplify data access and manipulation by providing a virtual layer over one or more indices ([#11957](https://github.com/opensearch-project/OpenSearch/pull/11957))
20-
20+
- Add Remote Store Migration Experimental flag and allow mixed mode clusters under same ([#11986](https://github.com/opensearch-project/OpenSearch/pull/11986))
2121

2222
### Dependencies
2323
- Bump `log4j-core` from 2.18.0 to 2.19.0
@@ -50,6 +50,9 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
5050
- Bump `org.eclipse.jgit` from 6.5.0 to 6.7.0 ([#10147](https://github.com/opensearch-project/OpenSearch/pull/10147))
5151
- Bump OpenTelemetry from 1.30.1 to 1.31.0 ([#10617](https://github.com/opensearch-project/OpenSearch/pull/10617))
5252
- Bump OpenTelemetry from 1.31.0 to 1.32.0 and OpenTelemetry Semconv from 1.21.0-alpha to 1.23.1-alpha ([#11305](https://github.com/opensearch-project/OpenSearch/pull/11305))
53+
- Bump `org.bouncycastle:bcprov-jdk15to18` to `org.bouncycastle:bcprov-jdk18on` version 1.77 ([#12317](https://github.com/opensearch-project/OpenSearch/pull/12317))
54+
- Bump `org.bouncycastle:bcmail-jdk15to18` to `org.bouncycastle:bcmail-jdk18on` version 1.77 ([#12317](https://github.com/opensearch-project/OpenSearch/pull/12317))
55+
- Bump `org.bouncycastle:bcpkix-jdk15to18` to `org.bouncycastle:bcpkix-jdk18on` version 1.77 ([#12317](https://github.com/opensearch-project/OpenSearch/pull/12317))
5356

5457
### Changed
5558
- [CCR] Add getHistoryOperationsFromTranslog method to fetch the history snapshot from translogs ([#3948](https://github.com/opensearch-project/OpenSearch/pull/3948))
@@ -89,6 +92,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
8992
- Update supported version for must_exist parameter in update aliases API ([#11872](https://github.com/opensearch-project/OpenSearch/pull/11872))
9093
- [Bug] Check phase name before SearchRequestOperationsListener onPhaseStart ([#12035](https://github.com/opensearch-project/OpenSearch/pull/12035))
9194
- Fix Span operation names generated from RestActions ([#12005](https://github.com/opensearch-project/OpenSearch/pull/12005))
95+
- Fix error in RemoteSegmentStoreDirectory when debug logging is enabled ([#12328](https://github.com/opensearch-project/OpenSearch/pull/12328))
9296

9397
### Security
9498

@@ -102,15 +106,19 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
102106
### Dependencies
103107
- Bump `peter-evans/find-comment` from 2 to 3 ([#12288](https://github.com/opensearch-project/OpenSearch/pull/12288))
104108
- Bump `com.google.api.grpc:proto-google-common-protos` from 2.25.1 to 2.33.0 ([#12289](https://github.com/opensearch-project/OpenSearch/pull/12289))
109+
- Bump `com.squareup.okio:okio` from 3.7.0 to 3.8.0 ([#12290](https://github.com/opensearch-project/OpenSearch/pull/12290))
105110

106111
### Changed
112+
- Allow composite aggregation to run under a parent filter aggregation ([#11499](https://github.com/opensearch-project/OpenSearch/pull/11499))
107113

108114
### Deprecated
109115

110116
### Removed
111117

112118
### Fixed
113119
- Fix for deserilization bug in weighted round-robin metadata ([#11679](https://github.com/opensearch-project/OpenSearch/pull/11679))
120+
- Add a system property to configure YamlParser codepoint limits ([#12298](https://github.com/opensearch-project/OpenSearch/pull/12298))
121+
- Prevent read beyond slice boundary in ByteArrayIndexInput ([#10481](https://github.com/opensearch-project/OpenSearch/issues/10481))
114122

115123
### Security
116124

buildSrc/version.properties

+1-1
Original file line numberDiff line numberDiff line change
@@ -50,7 +50,7 @@ reactivestreams = 1.0.4
5050
# when updating this version, you need to ensure compatibility with:
5151
# - plugins/ingest-attachment (transitive dependency, check the upstream POM)
5252
# - distribution/tools/plugin-cli
53-
bouncycastle=1.76
53+
bouncycastle=1.77
5454
# test dependencies
5555
randomizedrunner = 2.7.1
5656
junit = 4.13.2

gradle/ide.gradle

+3
Original file line numberDiff line numberDiff line change
@@ -82,6 +82,9 @@ if (System.getProperty('idea.active') == 'true') {
8282
runConfigurations {
8383
defaults(JUnit) {
8484
vmParameters = '-ea -Djava.locale.providers=SPI,COMPAT'
85+
if (BuildParams.runtimeJavaVersion > JavaVersion.VERSION_17) {
86+
vmParameters += ' -Djava.security.manager=allow'
87+
}
8588
}
8689
}
8790
copyright {

libs/x-content/src/main/java/org/opensearch/common/xcontent/XContentContraints.java

+3
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@
1919
*/
2020
@InternalApi
2121
public interface XContentContraints {
22+
final String DEFAULT_CODEPOINT_LIMIT_PROPERTY = "opensearch.xcontent.codepoint.max";
2223
final String DEFAULT_MAX_STRING_LEN_PROPERTY = "opensearch.xcontent.string.length.max";
2324
final String DEFAULT_MAX_NAME_LEN_PROPERTY = "opensearch.xcontent.name.length.max";
2425
final String DEFAULT_MAX_DEPTH_PROPERTY = "opensearch.xcontent.depth.max";
@@ -32,4 +33,6 @@ public interface XContentContraints {
3233
final int DEFAULT_MAX_DEPTH = Integer.parseInt(
3334
System.getProperty(DEFAULT_MAX_DEPTH_PROPERTY, "1000" /* StreamReadConstraints.DEFAULT_MAX_DEPTH */)
3435
);
36+
37+
final int DEFAULT_CODEPOINT_LIMIT = Integer.parseInt(System.getProperty(DEFAULT_CODEPOINT_LIMIT_PROPERTY, "52428800" /* ~50 Mb */));
3538
}

libs/x-content/src/main/java/org/opensearch/common/xcontent/yaml/YamlXContent.java

+6-1
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,7 @@
3838
import com.fasterxml.jackson.core.StreamReadFeature;
3939
import com.fasterxml.jackson.core.StreamWriteConstraints;
4040
import com.fasterxml.jackson.dataformat.yaml.YAMLFactory;
41+
import com.fasterxml.jackson.dataformat.yaml.YAMLFactoryBuilder;
4142

4243
import org.opensearch.common.xcontent.XContentContraints;
4344
import org.opensearch.common.xcontent.XContentType;
@@ -55,6 +56,8 @@
5556
import java.io.Reader;
5657
import java.util.Set;
5758

59+
import org.yaml.snakeyaml.LoaderOptions;
60+
5861
/**
5962
* A YAML based content implementation using Jackson.
6063
*/
@@ -67,7 +70,9 @@ public static XContentBuilder contentBuilder() throws IOException {
6770
public static final YamlXContent yamlXContent;
6871

6972
static {
70-
yamlFactory = new YAMLFactory();
73+
final LoaderOptions loaderOptions = new LoaderOptions();
74+
loaderOptions.setCodePointLimit(DEFAULT_CODEPOINT_LIMIT);
75+
yamlFactory = new YAMLFactoryBuilder(new YAMLFactory()).loaderOptions(loaderOptions).build();
7176
yamlFactory.configure(JsonParser.Feature.STRICT_DUPLICATE_DETECTION, true);
7277
yamlFactory.setStreamWriteConstraints(StreamWriteConstraints.builder().maxNestingDepth(DEFAULT_MAX_DEPTH).build());
7378
yamlFactory.setStreamReadConstraints(

libs/x-content/src/test/java/org/opensearch/common/xcontent/XContentParserTests.java

+2-2
Original file line numberDiff line numberDiff line change
@@ -85,7 +85,7 @@ public class XContentParserTests extends OpenSearchTestCase {
8585
() -> randomAlphaOfLengthBetween(1, SmileXContent.DEFAULT_MAX_STRING_LEN),
8686
/* YAML parser limitation */
8787
XContentType.YAML,
88-
() -> randomAlphaOfLengthBetween(1, 3140000)
88+
() -> randomRealisticUnicodeOfCodepointLengthBetween(1, YamlXContent.DEFAULT_CODEPOINT_LIMIT)
8989
);
9090

9191
private static final Map<XContentType, Supplier<String>> OFF_LIMIT_GENERATORS = Map.of(
@@ -97,7 +97,7 @@ public class XContentParserTests extends OpenSearchTestCase {
9797
() -> randomAlphaOfLength(SmileXContent.DEFAULT_MAX_STRING_LEN + 1),
9898
/* YAML parser limitation */
9999
XContentType.YAML,
100-
() -> randomRealisticUnicodeOfCodepointLength(3145730)
100+
() -> randomRealisticUnicodeOfCodepointLength(YamlXContent.DEFAULT_CODEPOINT_LIMIT + 1)
101101
);
102102

103103
private static final Map<XContentType, Supplier<String>> FIELD_NAME_GENERATORS = Map.of(

plugins/identity-shiro/build.gradle

+1-1
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@ dependencies {
2828

2929
implementation 'org.passay:passay:1.6.3'
3030

31-
implementation "org.bouncycastle:bcprov-jdk15to18:${versions.bouncycastle}"
31+
implementation "org.bouncycastle:bcprov-jdk18on:${versions.bouncycastle}"
3232

3333
testImplementation project(path: ':modules:transport-netty4') // for http
3434
testImplementation project(path: ':plugins:transport-nio') // for http

plugins/identity-shiro/licenses/bcprov-jdk15to18-1.76.jar.sha1

-1
This file was deleted.
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
2cc971b6c20949c1ff98d1a4bc741ee848a09523

plugins/ingest-attachment/build.gradle

+3-3
Original file line numberDiff line numberDiff line change
@@ -71,9 +71,9 @@ dependencies {
7171
api "org.apache.pdfbox:fontbox:${versions.pdfbox}"
7272
api "org.apache.pdfbox:jempbox:1.8.17"
7373
api "commons-logging:commons-logging:${versions.commonslogging}"
74-
api "org.bouncycastle:bcmail-jdk15to18:${versions.bouncycastle}"
75-
api "org.bouncycastle:bcprov-jdk15to18:${versions.bouncycastle}"
76-
api "org.bouncycastle:bcpkix-jdk15to18:${versions.bouncycastle}"
74+
api "org.bouncycastle:bcmail-jdk18on:${versions.bouncycastle}"
75+
api "org.bouncycastle:bcprov-jdk18on:${versions.bouncycastle}"
76+
api "org.bouncycastle:bcpkix-jdk18on:${versions.bouncycastle}"
7777
// OpenOffice
7878
api "org.apache.poi:poi-ooxml:${versions.poi}"
7979
api "org.apache.poi:poi:${versions.poi}"

plugins/ingest-attachment/licenses/bcmail-jdk15to18-1.76.jar.sha1

-1
This file was deleted.
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
f2bb8aa55dc901ee8b8aae7d1007c03592d65e03

plugins/ingest-attachment/licenses/bcpkix-jdk15to18-1.76.jar.sha1

-1
This file was deleted.
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
ed953791ba0229747dd0fd9911e3d76a462acfd3

plugins/ingest-attachment/licenses/bcprov-jdk15to18-1.76.jar.sha1

-1
This file was deleted.
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
2cc971b6c20949c1ff98d1a4bc741ee848a09523

release-notes/opensearch.release-notes-2.12.0.md

-1
Original file line numberDiff line numberDiff line change
@@ -128,7 +128,6 @@
128128
- Change error message when per shard document limit is breached ([#11312](https://github.com/opensearch-project/OpenSearch/pull/11312))
129129
- Improve boolean parsing performance ([#11308](https://github.com/opensearch-project/OpenSearch/pull/11308))
130130
- Interpret byte array as primitive using VarHandles ([#11362](https://github.com/opensearch-project/OpenSearch/pull/11362))
131-
- Allow composite aggregation to run under a parent filter aggregation ([#11499](https://github.com/opensearch-project/OpenSearch/pull/11499))
132131
- Automatically add scheme to discovery.ec2.endpoint ([#11512](https://github.com/opensearch-project/OpenSearch/pull/11512))
133132
- Restore support for Java 8 for RestClient ([#11562](https://github.com/opensearch-project/OpenSearch/pull/11562))
134133
- Add deleted doc count in _cat/shards ([#11678](https://github.com/opensearch-project/OpenSearch/pull/11678))
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
/*
2+
* SPDX-License-Identifier: Apache-2.0
3+
*
4+
* The OpenSearch Contributors require contributions made to
5+
* this file be licensed under the Apache-2.0 license or a
6+
* compatible open source license.
7+
*/
8+
9+
package org.opensearch.remotemigration;
10+
11+
import org.opensearch.action.admin.cluster.settings.ClusterUpdateSettingsRequest;
12+
import org.opensearch.client.Client;
13+
import org.opensearch.common.settings.Settings;
14+
import org.opensearch.test.OpenSearchIntegTestCase;
15+
16+
import java.util.List;
17+
18+
import static org.opensearch.node.remotestore.RemoteStoreNodeService.REMOTE_STORE_COMPATIBILITY_MODE_SETTING;
19+
import static org.opensearch.test.hamcrest.OpenSearchAssertions.assertAcked;
20+
21+
@OpenSearchIntegTestCase.ClusterScope(scope = OpenSearchIntegTestCase.Scope.TEST, numDataNodes = 0, autoManageMasterNodes = false)
22+
public class DocRepMigrationTestCase extends MigrationBaseTestCase {
23+
24+
public void testMixedModeAddDocRep() throws Exception {
25+
internalCluster().setBootstrapClusterManagerNodeIndex(0);
26+
List<String> cmNodes = internalCluster().startNodes(1);
27+
28+
Client client = internalCluster().client(cmNodes.get(0));
29+
ClusterUpdateSettingsRequest updateSettingsRequest = new ClusterUpdateSettingsRequest();
30+
updateSettingsRequest.persistentSettings(Settings.builder().put(REMOTE_STORE_COMPATIBILITY_MODE_SETTING.getKey(), "mixed"));
31+
assertAcked(client().admin().cluster().updateSettings(updateSettingsRequest).actionGet());
32+
addRemote = false;
33+
internalCluster().startNode();
34+
String[] allNodes = internalCluster().getNodeNames();
35+
assertBusy(() -> { assertEquals(client.admin().cluster().prepareClusterStats().get().getNodes().size(), allNodes.length); });
36+
}
37+
38+
}
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
/*
2+
* SPDX-License-Identifier: Apache-2.0
3+
*
4+
* The OpenSearch Contributors require contributions made to
5+
* this file be licensed under the Apache-2.0 license or a
6+
* compatible open source license.
7+
*/
8+
9+
package org.opensearch.remotemigration;
10+
11+
import org.opensearch.common.settings.Settings;
12+
import org.opensearch.common.util.FeatureFlags;
13+
import org.opensearch.test.OpenSearchIntegTestCase;
14+
15+
import java.nio.file.Path;
16+
17+
import static org.opensearch.remotestore.RemoteStoreBaseIntegTestCase.remoteStoreClusterSettings;
18+
19+
public class MigrationBaseTestCase extends OpenSearchIntegTestCase {
20+
protected static final String REPOSITORY_NAME = "test-remote-store-repo";
21+
protected static final String REPOSITORY_2_NAME = "test-remote-store-repo-2";
22+
23+
protected Path segmentRepoPath;
24+
protected Path translogRepoPath;
25+
26+
boolean addRemote = false;
27+
28+
protected Settings nodeSettings(int nodeOrdinal) {
29+
if (segmentRepoPath == null || translogRepoPath == null) {
30+
segmentRepoPath = randomRepoPath().toAbsolutePath();
31+
translogRepoPath = randomRepoPath().toAbsolutePath();
32+
}
33+
if (addRemote) {
34+
logger.info("Adding remote store node");
35+
return Settings.builder()
36+
.put(super.nodeSettings(nodeOrdinal))
37+
.put(remoteStoreClusterSettings(REPOSITORY_NAME, segmentRepoPath, REPOSITORY_2_NAME, translogRepoPath))
38+
.put("discovery.initial_state_timeout", "500ms")
39+
.build();
40+
} else {
41+
logger.info("Adding docrep node");
42+
return Settings.builder().put(super.nodeSettings(nodeOrdinal)).put("discovery.initial_state_timeout", "500ms").build();
43+
}
44+
}
45+
46+
@Override
47+
protected Settings featureFlagSettings() {
48+
return Settings.builder().put(super.featureFlagSettings()).put(FeatureFlags.REMOTE_STORE_MIGRATION_EXPERIMENTAL, "true").build();
49+
}
50+
}
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
/*
2+
* SPDX-License-Identifier: Apache-2.0
3+
*
4+
* The OpenSearch Contributors require contributions made to
5+
* this file be licensed under the Apache-2.0 license or a
6+
* compatible open source license.
7+
*/
8+
9+
package org.opensearch.remotemigration;
10+
11+
import org.opensearch.action.admin.cluster.repositories.get.GetRepositoriesRequest;
12+
import org.opensearch.action.admin.cluster.repositories.get.GetRepositoriesResponse;
13+
import org.opensearch.action.admin.cluster.settings.ClusterUpdateSettingsRequest;
14+
import org.opensearch.client.Client;
15+
import org.opensearch.common.settings.Settings;
16+
import org.opensearch.test.OpenSearchIntegTestCase;
17+
18+
import java.util.List;
19+
20+
import static org.opensearch.node.remotestore.RemoteStoreNodeService.MIGRATION_DIRECTION_SETTING;
21+
import static org.opensearch.node.remotestore.RemoteStoreNodeService.REMOTE_STORE_COMPATIBILITY_MODE_SETTING;
22+
import static org.opensearch.remotestore.RemoteStoreBaseIntegTestCase.remoteStoreClusterSettings;
23+
import static org.opensearch.test.hamcrest.OpenSearchAssertions.assertAcked;
24+
25+
@OpenSearchIntegTestCase.ClusterScope(scope = OpenSearchIntegTestCase.Scope.TEST, numDataNodes = 0, autoManageMasterNodes = false)
26+
public class RemoteStoreMigrationTestCase extends MigrationBaseTestCase {
27+
public void testMixedModeAddRemoteNodes() throws Exception {
28+
internalCluster().setBootstrapClusterManagerNodeIndex(0);
29+
List<String> cmNodes = internalCluster().startNodes(1);
30+
Client client = internalCluster().client(cmNodes.get(0));
31+
ClusterUpdateSettingsRequest updateSettingsRequest = new ClusterUpdateSettingsRequest();
32+
updateSettingsRequest.persistentSettings(Settings.builder().put(REMOTE_STORE_COMPATIBILITY_MODE_SETTING.getKey(), "mixed"));
33+
assertAcked(client().admin().cluster().updateSettings(updateSettingsRequest).actionGet());
34+
35+
// add remote node in mixed mode cluster
36+
addRemote = true;
37+
internalCluster().startNode();
38+
internalCluster().startNode();
39+
internalCluster().validateClusterFormed();
40+
41+
// assert repo gets registered
42+
GetRepositoriesRequest gr = new GetRepositoriesRequest(new String[] { REPOSITORY_NAME });
43+
GetRepositoriesResponse getRepositoriesResponse = client.admin().cluster().getRepositories(gr).actionGet();
44+
assertEquals(1, getRepositoriesResponse.repositories().size());
45+
46+
// add docrep mode in mixed mode cluster
47+
addRemote = true;
48+
internalCluster().startNode();
49+
assertBusy(() -> {
50+
assertEquals(client.admin().cluster().prepareClusterStats().get().getNodes().size(), internalCluster().getNodeNames().length);
51+
});
52+
53+
// add incompatible remote node in remote mixed cluster
54+
Settings.Builder badSettings = Settings.builder()
55+
.put(remoteStoreClusterSettings(REPOSITORY_NAME, segmentRepoPath, "REPOSITORY_2_NAME", translogRepoPath))
56+
.put("discovery.initial_state_timeout", "500ms");
57+
String badNode = internalCluster().startNode(badSettings);
58+
assertTrue(client.admin().cluster().prepareClusterStats().get().getNodes().size() < internalCluster().getNodeNames().length);
59+
internalCluster().stopRandomNode(settings -> settings.get("node.name").equals(badNode));
60+
}
61+
62+
public void testMigrationDirections() {
63+
ClusterUpdateSettingsRequest updateSettingsRequest = new ClusterUpdateSettingsRequest();
64+
// add remote node in docrep cluster
65+
updateSettingsRequest.persistentSettings(Settings.builder().put(MIGRATION_DIRECTION_SETTING.getKey(), "docrep"));
66+
assertAcked(client().admin().cluster().updateSettings(updateSettingsRequest).actionGet());
67+
68+
updateSettingsRequest.persistentSettings(Settings.builder().put(MIGRATION_DIRECTION_SETTING.getKey(), "remote_store"));
69+
assertAcked(client().admin().cluster().updateSettings(updateSettingsRequest).actionGet());
70+
71+
updateSettingsRequest.persistentSettings(Settings.builder().put(MIGRATION_DIRECTION_SETTING.getKey(), "random"));
72+
assertThrows(IllegalArgumentException.class, () -> client().admin().cluster().updateSettings(updateSettingsRequest).actionGet());
73+
}
74+
}

0 commit comments

Comments
 (0)