Skip to content

Commit f965695

Browse files
DarshitChanpuraDarshitChanpura
committed
Merge remote-tracking branch 'upstream/main' into intro-resource-permissions
2 parents a5ded4b + 88cd4ee commit f965695

File tree

234 files changed

+1238
-375
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

234 files changed

+1238
-375
lines changed

‎.github/workflows/changelog_verifier.yml

+23
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
name: "Changelog Verifier"
2+
on:
3+
push:
4+
branches-ignore:
5+
- 'whitesource-remediate/**'
6+
- 'backport/**'
7+
pull_request:
8+
types: [opened, edited, review_requested, synchronize, reopened, ready_for_review, labeled, unlabeled]
9+
10+
jobs:
11+
# Enforces the update of a changelog file on every pull request
12+
verify-changelog:
13+
if: github.repository == 'opensearch-project/anomaly-detection'
14+
runs-on: ubuntu-latest
15+
steps:
16+
- uses: actions/checkout@v4
17+
with:
18+
token: ${{ secrets.GITHUB_TOKEN }}
19+
ref: ${{ github.event.pull_request.head.sha }}
20+
21+
- uses: dangoslen/changelog-enforcer@v3
22+
with:
23+
skipLabels: "autocut, skip-changelog"

‎.github/workflows/test_build_multi_platform.yml

+3-3
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@ jobs:
2929
needs: spotless
3030
strategy:
3131
matrix:
32-
java: [ 21 ]
32+
java: [21, 23]
3333
name: Build and Test Anomaly Detection Plugin on Windows
3434
runs-on: windows-latest
3535
env:
@@ -57,7 +57,7 @@ jobs:
5757
needs: [Get-CI-Image-Tag, spotless]
5858
strategy:
5959
matrix:
60-
java: [21]
60+
java: [21, 23]
6161
fail-fast: false
6262
name: Build and Test Anomaly detection Plugin on Linux
6363
runs-on: ubuntu-latest
@@ -95,7 +95,7 @@ jobs:
9595
needs: spotless
9696
strategy:
9797
matrix:
98-
java: [21]
98+
java: [21, 23]
9999
fail-fast: false
100100

101101
name: Build and Test Anomaly detection Plugin on MacOS

‎CHANGELOG.md

+30
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
# CHANGELOG
2+
All notable changes to this project are documented in this file.
3+
4+
Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.1.0/)
5+
6+
## [Unreleased 3.0](https://github.com/opensearch-project/anomaly-detection/compare/2.x...HEAD)
7+
### Features
8+
### Enhancements
9+
### Bug Fixes
10+
### Infrastructure
11+
12+
### Documentation
13+
14+
### Maintenance
15+
- Fix breaking changes for 3.0.0 release ([#1424](https://github.com/opensearch-project/anomaly-detection/pull/1424))
16+
17+
### Refactoring
18+
19+
## [Unreleased 2.x](https://github.com/opensearch-project/anomaly-detection/compare/2.19...2.x)
20+
### Features
21+
22+
23+
### Enhancements
24+
- Github workflow for changelog verification ([#1413](https://github.com/opensearch-project/anomaly-detection/pull/1413))
25+
### Bug Fixes
26+
27+
### Infrastructure
28+
### Documentation
29+
### Maintenance
30+
### Refactoring

‎DEVELOPER_GUIDE.md

+24
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@
1111
- [Backports](#backports)
1212
- [Gradle Plugins](#gradle-plugins)
1313
- [Distribution Download Plugin](#distribution-download-plugin)
14+
- [Changelog](#changelog)
1415

1516
## Developer Guide
1617

@@ -102,3 +103,26 @@ The Distribution Download plugin downloads the latest version of OpenSearch by d
102103
```
103104
./gradlew integTest -PcustomDistributionUrl="https://ci.opensearch.org/ci/dbc/bundle-build/1.2.0/1127/linux/x64/dist/opensearch-1.2.0-linux-x64.tar.gz"
104105
```
106+
107+
## Changelog
108+
109+
AD maintains version specific changelog by enforcing a change to the ongoing [CHANGELOG](CHANGELOG.md) file adhering to the [Keep A Changelog](https://keepachangelog.com/en/1.1.0/) format.
110+
111+
Briefly, the changes are curated by version, with the changes to the main branch added chronologically to `Unreleased` version. Further, each version has corresponding sections which list out the category of the change - `Added`, `Changed`, `Deprecated`, `Removed`, `Fixed`, `Security`.
112+
113+
#### How to add my changes to [CHANGELOG](CHANGELOG.md)?
114+
115+
As a contributor, you must ensure that every pull request has the changes listed out within the corresponding version and appropriate section of [CHANGELOG](CHANGELOG.md) file.
116+
117+
Adding in the change is two step process -
118+
1. Add your changes to the corresponding section within the CHANGELOG file with dummy pull request information, publish the PR
119+
120+
`Your change here ([#PR_NUMBER](PR_URL))`
121+
122+
2. Update the entry for your change in [`CHANGELOG.md`](CHANGELOG.md) and make sure that you reference the pull request there.
123+
124+
[Example PR](https://github.com/opensearch-project/flow-framework/pull/998/files#diff-06572a96a58dc510037d5efa622f9bec8519bc1beab13c9f251e97e657a9d4edR24)
125+
126+
For future release notes, all entries can be directly copied to the release notes and then deleted from either the 2.x or 3.0 section as a cleanup. For example, see: https://github.com/opensearch-project/flow-framework/pull/1036/files
127+
128+
For changes that don't require an entry, we can add the 'skip-changelog' label to the PR. This will allow the changelog workflow to pass without adding an entry."

‎build-tools/coverage.gradle

+1-1
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@
66
apply plugin: 'jacoco'
77

88
jacoco {
9-
toolVersion = "0.8.10"
9+
toolVersion = "0.8.12"
1010
}
1111

1212
/**

‎build.gradle

+4-4
Original file line numberDiff line numberDiff line change
@@ -18,8 +18,8 @@ buildscript {
1818
ext {
1919
opensearch_group = "org.opensearch"
2020
isSnapshot = "true" == System.getProperty("build.snapshot", "true")
21-
opensearch_version = System.getProperty("opensearch.version", "3.0.0-SNAPSHOT")
22-
buildVersionQualifier = System.getProperty("build.version_qualifier", "")
21+
opensearch_version = System.getProperty("opensearch.version", "3.0.0-alpha1-SNAPSHOT")
22+
buildVersionQualifier = System.getProperty("build.version_qualifier", "alpha1")
2323
// 3.0.0-SNAPSHOT -> 3.0.0.0-SNAPSHOT
2424
version_tokens = opensearch_version.tokenize('-')
2525
opensearch_build = version_tokens[0] + '.0'
@@ -35,7 +35,7 @@ buildscript {
3535
js_resource_folder = "src/test/resources/job-scheduler"
3636
common_utils_version = System.getProperty("common_utils.version", opensearch_build)
3737
job_scheduler_version = System.getProperty("job_scheduler.version", opensearch_build)
38-
bwcVersionShort = "2.19.0"
38+
bwcVersionShort = "2.20.0"
3939
bwcVersion = bwcVersionShort + ".0"
4040
bwcOpenSearchADDownload = 'https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/' + bwcVersionShort + '/latest/linux/x64/tar/builds/' +
4141
'opensearch/plugins/opensearch-anomaly-detection-' + bwcVersion + '.zip'
@@ -65,7 +65,7 @@ buildscript {
6565
}
6666

6767
plugins {
68-
id 'com.netflix.nebula.ospackage' version "11.5.0"
68+
id 'com.netflix.nebula.ospackage' version "11.10.0"
6969
id "com.diffplug.spotless" version "6.25.0"
7070
id 'java-library'
7171
id 'org.gradle.test-retry' version '1.6.0'

‎gradle/wrapper/gradle-wrapper.properties

+2-1
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
distributionBase=GRADLE_USER_HOME
22
distributionPath=wrapper/dists
3-
distributionUrl=https\://services.gradle.org/distributions/gradle-8.5-bin.zip
3+
distributionSha256Sum=2ab88d6de2c23e6adae7363ae6e29cbdd2a709e992929b48b6530fd0c7133bd6
4+
distributionUrl=https\://services.gradle.org/distributions/gradle-8.10.2-all.zip
45
networkTimeout=10000
56
validateDistributionUrl=true
67
zipStoreBase=GRADLE_USER_HOME

‎release-notes/opensearch-anomaly-detection.release-notes-2.19.0.0.md

+23
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
## Version 2.19.0.0 Release Notes
2+
3+
Compatible with OpenSearch 2.19.0
4+
5+
### Features
6+
* Allow triggering anomaly only on drop or rise of features ([#1358](https://github.com/opensearch-project/anomaly-detection/pull/1358))
7+
* Add flattens custom result index when enabled ([#1401](https://github.com/opensearch-project/anomaly-detection/pull/1401)), ([#1409](https://github.com/opensearch-project/anomaly-detection/pull/1409))
8+
9+
### Enhancements
10+
* Changing replica count to up 2 for custom result index ([#1362](https://github.com/opensearch-project/anomaly-detection/pull/1362))
11+
12+
### Bug Fixes
13+
* Not blocking detector creation on unknown feature validation error ([#1366](https://github.com/opensearch-project/anomaly-detection/pull/1366))
14+
* Fix exceptions in IntervalCalculation and ResultIndexingHandler ([#1379](https://github.com/opensearch-project/anomaly-detection/pull/1379))
15+
16+
### Infrastructure
17+
* Bump codecov/codecov-action from 4 to 5 ([#1369](https://github.com/opensearch-project/anomaly-detection/pull/1369))
18+
* Bump com.google.code.gson:gson from 2.8.9 to 2.11.0 ([#1375](https://github.com/opensearch-project/anomaly-detection/pull/1375))
19+
* Bump jackson from 2.18.0 to 2.18.2 ([#1376](https://github.com/opensearch-project/anomaly-detection/pull/1376))
20+
* Bump org.apache.commons:commons-lang3 from 3.13.0 to 3.17.0 ([#1377](https://github.com/opensearch-project/anomaly-detection/pull/1377))
21+
* Bump org.objenesis:objenesis from 3.3 to 3.4 ([#1393](https://github.com/opensearch-project/anomaly-detection/pull/1393))
22+
* Updating several dependencies ([#1368](https://github.com/opensearch-project/anomaly-detection/pull/1368))
23+
* Update recency_emphasis to be greater than 1 in test cases ([#1406](https://github.com/opensearch-project/anomaly-detection/pull/1406))

‎src/main/java/org/opensearch/ad/ADEntityProfileRunner.java

+1-1
Original file line numberDiff line numberDiff line change
@@ -16,11 +16,11 @@
1616
import org.opensearch.ad.model.AnomalyResult;
1717
import org.opensearch.ad.settings.ADNumericSetting;
1818
import org.opensearch.ad.transport.ADEntityProfileAction;
19-
import org.opensearch.client.Client;
2019
import org.opensearch.core.xcontent.NamedXContentRegistry;
2120
import org.opensearch.timeseries.AnalysisType;
2221
import org.opensearch.timeseries.EntityProfileRunner;
2322
import org.opensearch.timeseries.util.SecurityClientUtil;
23+
import org.opensearch.transport.client.Client;
2424

2525
public class ADEntityProfileRunner extends EntityProfileRunner<ADEntityProfileAction> {
2626

‎src/main/java/org/opensearch/ad/ADTaskProfileRunner.java

+1-1
Original file line numberDiff line numberDiff line change
@@ -15,11 +15,11 @@
1515
import org.opensearch.ad.transport.ADTaskProfileAction;
1616
import org.opensearch.ad.transport.ADTaskProfileNodeResponse;
1717
import org.opensearch.ad.transport.ADTaskProfileRequest;
18-
import org.opensearch.client.Client;
1918
import org.opensearch.core.action.ActionListener;
2019
import org.opensearch.timeseries.TaskProfileRunner;
2120
import org.opensearch.timeseries.cluster.HashRing;
2221
import org.opensearch.timeseries.model.EntityTaskProfile;
22+
import org.opensearch.transport.client.Client;
2323

2424
public class ADTaskProfileRunner implements TaskProfileRunner<ADTask, ADTaskProfile> {
2525
public final Logger logger = LogManager.getLogger(ADTaskProfileRunner.class);

‎src/main/java/org/opensearch/ad/AnomalyDetectorProfileRunner.java

+1-1
Original file line numberDiff line numberDiff line change
@@ -24,14 +24,14 @@
2424
import org.opensearch.ad.task.ADTaskCacheManager;
2525
import org.opensearch.ad.task.ADTaskManager;
2626
import org.opensearch.ad.transport.ADProfileAction;
27-
import org.opensearch.client.Client;
2827
import org.opensearch.core.xcontent.NamedXContentRegistry;
2928
import org.opensearch.timeseries.AnalysisType;
3029
import org.opensearch.timeseries.ProfileRunner;
3130
import org.opensearch.timeseries.model.ProfileName;
3231
import org.opensearch.timeseries.util.DiscoveryNodeFilterer;
3332
import org.opensearch.timeseries.util.SecurityClientUtil;
3433
import org.opensearch.transport.TransportService;
34+
import org.opensearch.transport.client.Client;
3535

3636
/**
3737
* Since version 2.15, we have merged the single-stream and HC detector workflows. Consequently, separate logic for profiling is no longer necessary.

‎src/main/java/org/opensearch/ad/ExecuteADResultResponseRecorder.java

+1-1
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,6 @@
2525
import org.opensearch.ad.task.ADTaskCacheManager;
2626
import org.opensearch.ad.task.ADTaskManager;
2727
import org.opensearch.ad.transport.ADProfileAction;
28-
import org.opensearch.client.Client;
2928
import org.opensearch.commons.authuser.User;
3029
import org.opensearch.threadpool.ThreadPool;
3130
import org.opensearch.timeseries.AnalysisType;
@@ -36,6 +35,7 @@
3635
import org.opensearch.timeseries.transport.ResultResponse;
3736
import org.opensearch.timeseries.transport.handler.ResultBulkIndexingHandler;
3837
import org.opensearch.timeseries.util.DiscoveryNodeFilterer;
38+
import org.opensearch.transport.client.Client;
3939

4040
public class ExecuteADResultResponseRecorder extends
4141
ExecuteResultResponseRecorder<ADIndex, ADIndexManagement, ADTaskCacheManager, ADTaskType, ADTask, ADTaskManager, AnomalyResult, ADProfileAction> {

‎src/main/java/org/opensearch/ad/client/AnomalyDetectionNodeClient.java

+1-1
Original file line numberDiff line numberDiff line change
@@ -13,11 +13,11 @@
1313
import org.opensearch.ad.transport.GetAnomalyDetectorResponse;
1414
import org.opensearch.ad.transport.SearchAnomalyDetectorAction;
1515
import org.opensearch.ad.transport.SearchAnomalyResultAction;
16-
import org.opensearch.client.Client;
1716
import org.opensearch.core.action.ActionListener;
1817
import org.opensearch.core.action.ActionResponse;
1918
import org.opensearch.core.common.io.stream.NamedWriteableRegistry;
2019
import org.opensearch.timeseries.transport.GetConfigRequest;
20+
import org.opensearch.transport.client.Client;
2121

2222
public class AnomalyDetectionNodeClient implements AnomalyDetectionClient {
2323
private final Client client;

‎src/main/java/org/opensearch/ad/indices/ADIndexManagement.java

+20-1
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@
2222

2323
import java.io.IOException;
2424
import java.util.EnumMap;
25+
import java.util.Map;
2526

2627
import org.apache.logging.log4j.LogManager;
2728
import org.apache.logging.log4j.Logger;
@@ -32,7 +33,6 @@
3233
import org.opensearch.ad.constant.ADCommonName;
3334
import org.opensearch.ad.model.AnomalyDetector;
3435
import org.opensearch.ad.model.AnomalyResult;
35-
import org.opensearch.client.Client;
3636
import org.opensearch.cluster.service.ClusterService;
3737
import org.opensearch.common.settings.Settings;
3838
import org.opensearch.common.xcontent.XContentType;
@@ -44,6 +44,9 @@
4444
import org.opensearch.timeseries.common.exception.EndRunException;
4545
import org.opensearch.timeseries.indices.IndexManagement;
4646
import org.opensearch.timeseries.util.DiscoveryNodeFilterer;
47+
import org.opensearch.transport.client.Client;
48+
49+
import com.fasterxml.jackson.databind.ObjectMapper;
4750

4851
/**
4952
* This class provides utility methods for various anomaly detection indices.
@@ -122,6 +125,22 @@ public static String getResultMappings() throws IOException {
122125
return getMappings(ANOMALY_RESULTS_INDEX_MAPPING_FILE);
123126
}
124127

128+
/**
129+
* Retrieves the JSON mapping for the flattened result index with the "dynamic" field set to true
130+
* @return JSON mapping for the flattened result index.
131+
* @throws IOException if the mapping file cannot be read.
132+
*/
133+
public static String getFlattenedResultMappings() throws IOException {
134+
ObjectMapper objectMapper = new ObjectMapper();
135+
136+
Map<String, Object> mapping = objectMapper
137+
.readValue(ADIndexManagement.class.getClassLoader().getResourceAsStream(ANOMALY_RESULTS_INDEX_MAPPING_FILE), Map.class);
138+
139+
mapping.put("dynamic", true);
140+
141+
return objectMapper.writeValueAsString(mapping);
142+
}
143+
125144
/**
126145
* Get anomaly detector state index mapping json content.
127146
*

‎src/main/java/org/opensearch/ad/ml/ADCheckpointDao.java

+1-1
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,6 @@
3838
import org.opensearch.ad.constant.ADCommonName;
3939
import org.opensearch.ad.indices.ADIndex;
4040
import org.opensearch.ad.indices.ADIndexManagement;
41-
import org.opensearch.client.Client;
4241
import org.opensearch.core.action.ActionListener;
4342
import org.opensearch.index.IndexNotFoundException;
4443
import org.opensearch.index.query.MatchQueryBuilder;
@@ -52,6 +51,7 @@
5251
import org.opensearch.timeseries.ml.SingleStreamModelIdMapper;
5352
import org.opensearch.timeseries.model.Entity;
5453
import org.opensearch.timeseries.util.ClientUtil;
54+
import org.opensearch.transport.client.Client;
5555

5656
import com.amazon.randomcutforest.RandomCutForest;
5757
import com.amazon.randomcutforest.config.Precision;

‎src/main/java/org/opensearch/ad/model/AnomalyDetector.java

+1-1
Original file line numberDiff line numberDiff line change
@@ -592,7 +592,7 @@ public static AnomalyDetector parse(
592592
case RESULT_INDEX_FIELD_TTL:
593593
customResultIndexTTL = onlyParseNumberValue(parser);
594594
break;
595-
case FLATTEN_RESULT_INDEX_MAPPING:
595+
case FLATTEN_CUSTOM_RESULT_INDEX:
596596
flattenResultIndexMapping = onlyParseBooleanValue(parser);
597597
break;
598598
case BREAKING_UI_CHANGE_TIME:

‎src/main/java/org/opensearch/ad/ratelimit/ADResultWriteRequest.java

+3-2
Original file line numberDiff line numberDiff line change
@@ -25,9 +25,10 @@ public ADResultWriteRequest(
2525
String detectorId,
2626
RequestPriority priority,
2727
AnomalyResult result,
28-
String resultIndex
28+
String resultIndex,
29+
String flattenResultIndex
2930
) {
30-
super(expirationEpochMs, detectorId, priority, result, resultIndex);
31+
super(expirationEpochMs, detectorId, priority, result, resultIndex, flattenResultIndex);
3132
}
3233

3334
public ADResultWriteRequest(StreamInput in) throws IOException {

‎src/main/java/org/opensearch/ad/ratelimit/ADResultWriteWorker.java

+3-2
Original file line numberDiff line numberDiff line change
@@ -103,8 +103,9 @@ protected ADResultWriteRequest createResultWriteRequest(
103103
String configId,
104104
RequestPriority priority,
105105
AnomalyResult result,
106-
String resultIndex
106+
String resultIndex,
107+
String flattenResultIndex
107108
) {
108-
return new ADResultWriteRequest(expirationEpochMs, configId, priority, result, resultIndex);
109+
return new ADResultWriteRequest(expirationEpochMs, configId, priority, result, resultIndex, flattenResultIndex);
109110
}
110111
}

‎src/main/java/org/opensearch/ad/ratelimit/ADSaveResultStrategy.java

+2-1
Original file line numberDiff line numberDiff line change
@@ -86,7 +86,8 @@ public void saveResult(AnomalyResult result, Config config) {
8686
config.getId(),
8787
result.getAnomalyGrade() > 0 ? RequestPriority.HIGH : RequestPriority.MEDIUM,
8888
result,
89-
config.getCustomResultIndexOrAlias()
89+
config.getCustomResultIndexOrAlias(),
90+
config.getFlattenResultIndexAlias()
9091
)
9192
);
9293
}

‎src/main/java/org/opensearch/ad/rest/RestAnomalyDetectorJobAction.java

+1-1
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,6 @@
2323
import org.opensearch.ad.constant.ADCommonMessages;
2424
import org.opensearch.ad.settings.ADEnabledSetting;
2525
import org.opensearch.ad.transport.AnomalyDetectorJobAction;
26-
import org.opensearch.client.node.NodeClient;
2726
import org.opensearch.cluster.service.ClusterService;
2827
import org.opensearch.common.settings.Settings;
2928
import org.opensearch.common.unit.TimeValue;
@@ -33,6 +32,7 @@
3332
import org.opensearch.timeseries.model.DateRange;
3433
import org.opensearch.timeseries.rest.RestJobAction;
3534
import org.opensearch.timeseries.transport.JobRequest;
35+
import org.opensearch.transport.client.node.NodeClient;
3636

3737
import com.google.common.collect.ImmutableList;
3838

‎src/main/java/org/opensearch/ad/rest/RestDeleteAnomalyDetectorAction.java

+1-1
Original file line numberDiff line numberDiff line change
@@ -20,12 +20,12 @@
2020
import org.opensearch.ad.constant.ADCommonMessages;
2121
import org.opensearch.ad.settings.ADEnabledSetting;
2222
import org.opensearch.ad.transport.DeleteAnomalyDetectorAction;
23-
import org.opensearch.client.node.NodeClient;
2423
import org.opensearch.rest.BaseRestHandler;
2524
import org.opensearch.rest.RestRequest;
2625
import org.opensearch.rest.action.RestToXContentListener;
2726
import org.opensearch.timeseries.TimeSeriesAnalyticsPlugin;
2827
import org.opensearch.timeseries.transport.DeleteConfigRequest;
28+
import org.opensearch.transport.client.node.NodeClient;
2929

3030
import com.google.common.collect.ImmutableList;
3131

0 commit comments

Comments
 (0)