AD Enhancements in Version 2.15

CONTRIBUTING.md

@@ -82,7 +82,7 @@ Follow these steps to set up your local copy of the repository:
 
    ```
    curl -sSL https://get.rvm.io | bash -s stable
-   rvm install 3.2
+   rvm install 3.2.4
    ruby -v
    ```
 

_observing-your-data/ad/index.md

@@ -30,7 +30,39 @@
    - Enter a name and brief description. Make sure the name is unique and descriptive enough to help you to identify the purpose of the detector.
 1. Specify the data source.   
    - For **Data source**, choose the index you want to use as the data source. You can optionally use index patterns to choose multiple indexes.
-   - (Optional) For **Data filter**, filter the index you chose as the data source. From the **Data filter** menu, choose **Add data filter**, and then design your filter query by selecting **Field**, **Operator**, and **Value**, or choose **Use query DSL** and add your own JSON filter query.
+   - (Optional) For **Data filter**, filter the index you chose as the data source. From the **Data filter** menu, choose **Add data filter**, and then design your filter query by selecting **Field**, **Operator**, and **Value**, or choose **Use query DSL** and add your own JSON filter query. Only a [Boolean query]({{site.url}}{{site.baseurl}}/query-dsl/compound/bool/) is supported for query domain-specific language (DSL).
+
+#### Example filter using query DSL
+The query is designed to retrieve documents where the `urlPath.keyword` field matches one of the following specified values:
+
+   - /domain/{id}/short
+   - /sub_dir/{id}/short
+   - /abcd/123/{id}/xyz
+
+   ```json
+   {
+      "bool": {
+         "should": [
+               {
+                  "term": {
+                     "urlPath.keyword": "/domain/{id}/short"
+                  }
+               },
+               {
+                  "term": {
+                     "urlPath.keyword": "/sub_dir/{id}/short"
+                  }
+               },
+               {
+                  "term": {
+                     "urlPath.keyword": "/abcd/123/{id}/xyz"
+                  }
+               }
+         ]
+      }
+   }
+   ```
+
 1. Specify a timestamp.    
    - Select the **Timestamp field** in your index.
 1. Define operation settings.
@@ -45,22 +77,33 @@
       - This value tells the detector that the data is not ingested into OpenSearch in real time but with a certain delay. Set the window delay to shift the detector interval to account for this delay.
       - For example, say the detector interval is 10 minutes and data is ingested into your cluster with a general delay of 1 minute. Assume the detector runs at 2:00. The detector attempts to get the last 10 minutes of data from 1:50 to 2:00, but because of the 1-minute delay, it only gets 9 minutes of data and misses the data from 1:59 to 2:00. Setting the window delay to 1 minute shifts the interval window to 1:49--1:59, so the detector accounts for all 10 minutes of the detector interval time.
 1. Specify custom result index.
-   - If you want to store the anomaly detection results in your own index, choose **Enable custom result index** and specify the custom index to store the result. The anomaly detection plugin adds an `opensearch-ad-plugin-result-` prefix to the index name that you input. For example, if you input `abc` as the result index name, the final index name is `opensearch-ad-plugin-result-abc`.
+   - The Anomaly Detection plugin allows you to store anomaly detection results in a custom index of your choice. To enable this, select **Enable custom result index** and provide a name for your index, for example, `abc`. The plugin then creates an alias prefixed with `opensearch-ad-plugin-result-` followed by your chosen name, for example, `opensearch-ad-plugin-result-abc`. This alias points to an actual index with a name containing the date and a sequence number, like `opensearch-ad-plugin-result-abc-history-2024.06.12-000002`, where your results are stored.
 
    You can use the dash “-” sign to separate the namespace to manage custom result index permissions. For example, if you use `opensearch-ad-plugin-result-financial-us-group1` as the result index, you can create a permission role based on the pattern `opensearch-ad-plugin-result-financial-us-*` to represent the "financial" department at a granular level for the "us" area.
    {: .note }
 
-      - If the custom index you specify doesn’t already exist, the Anomaly Detection plugin creates this index when you create the detector and start your real-time or historical analysis.
-      - If the custom index already exists, the plugin checks if the index mapping of the custom index matches the anomaly result file. You need to make sure the custom index has valid mapping as shown here: [anomaly-results.json](https://github.com/opensearch-project/anomaly-detection/blob/main/src/main/resources/mappings/anomaly-results.json).
+   - When the Security plugin (fine-grained access control) is enabled, the default result index becomes a system index and is no longer accessible through the standard Index or Search API. To access its content, you must use the anomaly detection RESTful API or the dashboard. As a result, you cannot build customized dashboards using the default result index if the Security plugin is enabled. However, you can create a custom result index to build customized dashboards.
+   - If the custom index you specify does not exist, the Anomaly Detection plugin will create it when you create the detector and start your real-time or historical analysis.
+   - If the custom index already exists, the plugin will verify that the index mapping matches the required structure for anomaly results. In this case, ensure that the custom index has a valid mapping as defined in the [`anomaly-results.json`](https://github.com/opensearch-project/anomaly-detection/blob/main/src/main/resources/mappings/anomaly-results.json) file.
    - To use the custom result index option, you need the following permissions:
-      - `indices:admin/create` - If the custom index already exists, you don't need this.
+      - `indices:admin/create` - The Anomaly Detection plugin requires the ability to create and roll over the custom index.
+      - `indices:admin/aliases` - The Anomaly Detection plugin requires access to create and manage an alias for the custom index.
       - `indices:data/write/index` - You need the `write` permission for the Anomaly Detection plugin to write results into the custom index for a single-entity detector.
       - `indices:data/read/search` - You need the `search` permission because the Anomaly Detection plugin needs to search custom result indexes to show results on the anomaly detection UI.
       - `indices:data/write/delete` - Because the detector might generate a large number of anomaly results, you need the `delete` permission to delete old data and save disk space.
       - `indices:data/write/bulk*` -  You need the `bulk*` permission because the Anomaly Detection plugin uses the bulk API to write results into the custom index.
    - Managing the custom result index:
       - The anomaly detection dashboard queries all detectors’ results from all custom result indexes. Having too many custom result indexes might impact the performance of the Anomaly Detection plugin.
       - You can use [Index State Management]({{site.url}}{{site.baseurl}}/im-plugin/ism/index/) to rollover old result indexes. You can also manually delete or archive any old result indexes. We recommend reusing a custom result index for multiple detectors.
+      - The Anomaly Detection plugin also provides lifecycle management for custom indexes. It rolls an alias over to a new index when the custom result index meets any of the following conditions:
+
+
+      Parameter | Description | Type | Unit | Example | Required
+      :--- | :--- |:--- |:--- |:--- |:---
+      `result_index_min_size` | The minimum total size of primary shards (excluding replicas) required for index rollover. If set to 100 GiB and the index has 5 primary and 5 replica shards of 20 GiB each, the total primary shard size is 100 GiB, triggering the rollover. | `integer` | `MB` | `51200` | No
+      `result_index_min_age` |  The minimum age of the index required for rollover, calculated from its creation time to the current time. | `integer` |`day` | `7` | No
+      `result_index_ttl` | The minimum age required to permanently delete rolled over indexes. | `integer` | `day` | `60` | No
+
 1. Choose **Next**.   
 
 After you define the detector, the next step is to configure the model.

_observing-your-data/ad/settings.md

@@ -49,4 +49,5 @@ plugins.anomaly_detection.dedicated_cache_size | 10 | If the real-time analysis
 plugins.anomaly_detection.max_concurrent_preview | 2 | The maximum number of concurrent previews. You can use this setting to limit resource usage.
 plugins.anomaly_detection.model_max_size_percent | 0.1 | The upper bound of the memory percentage for a model.
 plugins.anomaly_detection.door_keeper_in_cache.enabled | False | When set to `true`, OpenSearch places a bloom filter in front of an inactive entity cache to filter out items that are not likely to appear more than once.
-plugins.anomaly_detection.hcad_cold_start_interpolation.enabled | False | When set to `true`, enables interpolation in high-cardinality anomaly detection (HCAD) cold start.
+plugins.anomaly_detection.hcad_cold_start_interpolation.enabled | False | When set to true, enables interpolation for high-cardinality anomaly detection (HCAD) during the initial cold start period.
+plugins.anomaly_detection.jvm_heap_usage_threshold | 95 | Specifies the JVM memory usage threshold, as a percentage, at which anomaly detectors will be disabled. The default value is 95%, meaning detectors will be disabled when JVM heap usage reaches 95%.