enabled mockTelemetryPlugin for IT and fixed OOM (opensearch-project#13054)

* Disable stackTrace in MockSpanData by default

Signed-off-by: Atharva Sharma <atharvasharma61@gmail.com>

* enabled MockTelemetryPlugin for ITs

Signed-off-by: Atharva Sharma <atharvasharma61@gmail.com>

* Added the flag as system property

Signed-off-by: Atharva Sharma <atharvasharma61@gmail.com>

* Applied java spotless check

Signed-off-by: Atharva Sharma <atharvasharma61@gmail.com>

* Added details in changelog

Signed-off-by: Atharva Sharma <atharvasharma61@gmail.com>

* Added details in TESTING.md

Signed-off-by: Atharva Sharma <atharvasharma61@gmail.com>

* Update TESTING.md

Signed-off-by: Atharva Sharma <60044988+atharvasharma61@users.noreply.github.com>

---------

Signed-off-by: Atharva Sharma <atharvasharma61@gmail.com>
Signed-off-by: Atharva Sharma <60044988+atharvasharma61@users.noreply.github.com>

Author: atharvasharma61

CHANGELOG.md

@@ -55,6 +55,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Fix issue with feature flags where default value may not be honored ([#12849](https://github.com/opensearch-project/OpenSearch/pull/12849))
 - Fix UOE While building Exists query for nested search_as_you_type field ([#12048](https://github.com/opensearch-project/OpenSearch/pull/12048))
 - Client with Java 8 runtime and Apache HttpClient 5 Transport fails with java.lang.NoSuchMethodError: java.nio.ByteBuffer.flip()Ljava/nio/ByteBuffer ([#13100](https://github.com/opensearch-project/opensearch-java/pull/13100))
+- Enabled mockTelemetryPlugin for IT and fixed OOM issues ([#13054](https://github.com/opensearch-project/OpenSearch/pull/13054))
 - Fix implement mark() and markSupported() in class FilterStreamInput ([#13098](https://github.com/opensearch-project/OpenSearch/pull/13098))
 - Fix snapshot _status API to return correct status for partial snapshots ([#12812](https://github.com/opensearch-project/OpenSearch/pull/12812))
 

TESTING.md

@@ -84,6 +84,7 @@ This will instruct all JVMs (including any that run cli tools such as creating t
 -   In order to remotely attach a debugger to the process: `--debug-jvm`
 -   In order to set a different keystore password: `--keystore-password yourpassword`
 -   In order to set an OpenSearch setting, provide a setting with the following prefix: `-Dtests.opensearch.`
+-   In order to enable stack trace of the MockSpanData during testing, add: `-Dtests.telemetry.span.stack_traces=true` (Storing stack traces alongside span data can be useful for comprehensive debugging and performance optimization during testing, as it provides insights into the exact code paths and execution sequences, facilitating efficient issue identification and resolution. Note: Enabling this might lead to OOM issues while running ITs)
 
 ## Test case filtering
 
@@ -412,8 +413,8 @@ Say you need to make a change to `main` and have a BWC layer in `5.x`. You will
 You may want to run BWC tests for a secure OpenSearch cluster. In order to do this, you will need to follow a few additional steps:
 
 1. Clone the OpenSearch Security repository from https://github.com/opensearch-project/security.
-2. Get both the old version of the Security plugin (the version you wish to come from) and the new version of the Security plugin (the version you wish to go to). This can be done either by fetching the maven artifact with a command like `wget https://repo1.maven.org/maven2/org/opensearch/plugin/opensearch-security/<TARGET_VERSION>.0/opensearch-security-<TARGET_VERSION>.0.zip` or by running `./gradlew assemble` from the base of the Security repository. 
-3. Move both of the Security artifacts into new directories at the path `/security/bwc-test/src/test/resources/<TARGET_VERSION>.0`. You should end up with two different directories in `/security/bwc-test/src/test/resources/`, one named the old version and one the new version. 
+2. Get both the old version of the Security plugin (the version you wish to come from) and the new version of the Security plugin (the version you wish to go to). This can be done either by fetching the maven artifact with a command like `wget https://repo1.maven.org/maven2/org/opensearch/plugin/opensearch-security/<TARGET_VERSION>.0/opensearch-security-<TARGET_VERSION>.0.zip` or by running `./gradlew assemble` from the base of the Security repository.
+3. Move both of the Security artifacts into new directories at the path `/security/bwc-test/src/test/resources/<TARGET_VERSION>.0`. You should end up with two different directories in `/security/bwc-test/src/test/resources/`, one named the old version and one the new version.
 4. Run the following command from the base of the Security repository:
 
 ```
@@ -428,7 +429,7 @@ You may want to run BWC tests for a secure OpenSearch cluster. In order to do th
 
 `-Dtests.security.manager=false` handles access issues when attempting to read the certificates from the file system.
 `-Dtests.opensearch.http.protocol=https` tells the wait for cluster startup task to do the right thing.
-`-PcustomDistributionUrl=...` uses a custom build of the distribution of OpenSearch. This is unnecessary when running against standard/unmodified OpenSearch core distributions. 
+`-PcustomDistributionUrl=...` uses a custom build of the distribution of OpenSearch. This is unnecessary when running against standard/unmodified OpenSearch core distributions.
 
 ### Skip fetching latest
 

test/framework/src/main/java/org/opensearch/test/OpenSearchIntegTestCase.java

@@ -2096,8 +2096,7 @@ protected boolean addMockGeoShapeFieldMapper() {
      * @return boolean.
      */
     protected boolean addMockTelemetryPlugin() {
-        // setting to false until https://github.com/opensearch-project/OpenSearch/issues/12615 is resolved
-        return false;
+        return true;
     }
 
     /**

test/telemetry/src/main/java/org/opensearch/test/telemetry/tracing/StrictCheckSpanProcessor.java

@@ -8,6 +8,7 @@
 
 package org.opensearch.test.telemetry.tracing;
 
+import org.opensearch.common.Booleans;
 import org.opensearch.telemetry.tracing.Span;
 import org.opensearch.test.telemetry.tracing.validators.AllSpansAreEndedProperly;
 import org.opensearch.test.telemetry.tracing.validators.AllSpansHaveUniqueId;
@@ -29,6 +30,14 @@ public StrictCheckSpanProcessor() {}
 
     private static Map<String, MockSpanData> spanMap = new ConcurrentHashMap<>();
 
+    // If you want to see the stack trace for each spanData, then
+    // update the flag to true or set the corresponding system property to true
+    // This is helpful in debugging the tests. Default value is false.
+    // Note: Enabling this might lead to OOM issues while running ITs.
+    private static final boolean isStackTraceForSpanEnabled = Booleans.parseBoolean(
+        System.getProperty("tests.telemetry.span.stack_traces", "false")
+    );
+
     @Override
     public void onStart(Span span) {
         spanMap.put(span.getSpanId(), toMockSpanData(span));
@@ -53,14 +62,15 @@ public List<MockSpanData> getFinishedSpanItems() {
 
     private MockSpanData toMockSpanData(Span span) {
         String parentSpanId = (span.getParentSpan() != null) ? span.getParentSpan().getSpanId() : "";
+        StackTraceElement[] stackTrace = isStackTraceForSpanEnabled ? Thread.currentThread().getStackTrace() : null;
         MockSpanData spanData = new MockSpanData(
             span.getSpanId(),
             parentSpanId,
             span.getTraceId(),
             System.nanoTime(),
             false,
             span.getSpanName(),
-            Thread.currentThread().getStackTrace(),
+            stackTrace,
             (span instanceof MockSpan) ? ((MockSpan) span).getAttributes() : Map.of()
         );
         return spanData;