More fixes for the integration tests (opensearch-project#317)

Signed-off-by: Govind Kamat <govkamat@amazon.com>

Author: gkamat

.ci/build.sh

@@ -85,15 +85,8 @@ function build_it {
   make install
   make precommit
 
-  if [[ "$1" == "3.8" ]]; then
-    make it38
-  elif [[ "$1" == "3.9" ]]; then
-    make it39
-  elif [[ "$1" == "3.10" ]]; then
-    make it310
-  elif [[ "$1" == "3.11" ]]; then
-    make it311
-  fi
+  # make it38, it39, etc.
+  make "it${1//./}"
 }
 
 function license-scan {

DEVELOPER_GUIDE.md

@@ -3,19 +3,29 @@
 This document will walk you through on what's needed to start contributing code to OpenSearch Benchmark.
 
 ## Installation
+
 ### Prerequisites
 
-- Pyenv : Install `pyenv` and follow the instructions in the output of `pyenv init` to setup your shell and restart it before proceeding.
-For more details please refer to the [PyEnv installation instructions](https://github.com/pyenv/pyenv#installation).
-- JDK : JDK version required to build OpenSearch. Please refer to the [build setup requirements](https://github.com/opensearch-project/OpenSearch/blob/ca564fd04f5059cf9e3ce8aba442575afb3d99f1/DEVELOPER_GUIDE.md#install-prerequisites).
-- Docker : Docker and additionally `docker-compose`  on Linux.
-- Git : git 1.9 or latter.
+  - Pyenv : Install `pyenv` and follow the instructions in the output of `pyenv init` to set up your shell and restart it before proceeding.
+    For more details please refer to the [PyEnv installation instructions](https://github.com/pyenv/pyenv#installation).
+
+  - JDK: Although OSB is a Python application, it optionally builds and provisions OpenSearch clusters.  JDK version 17 is used to build the current version of OpenSearch.  Please refer to the [build setup requirements](https://github.com/opensearch-project/OpenSearch/blob/ca564fd04f5059cf9e3ce8aba442575afb3d99f1/DEVELOPER_GUIDE.md#install-prerequisites).
+    Note that the `javadoc` executable should be available in the JDK installation.  An earlier version of the JDK can be used, but not all the integration tests will pass.
+
+    ```
+    export JAVA_HOME=/path/to/JDK17
+
+    ```
+
+  - Install Docker and `docker-compose`.  Start the Docker server.  The user running the integration tests should have the permissions required to run docker commands.  Test by running `docker ps`.
+
+  - Git : git 1.9 or later.
 
 ### Setup
 
-Use the following command-line instructions to setup OpenSearch Benchmark for development:
+Use the following command-line instructions to set up OpenSearch Benchmark for development:
 ```
-git clone https://github.com/opensearch-project/OpenSearch-Benchmark.git
+git clone https://github.com/opensearch-project/OpenSearch-Benchmark
 cd OpenSearch-Benchmark
 make prereq
 make install
@@ -52,56 +62,29 @@ In order to run tests within the PyCharm IDE, ensure the `Python Integrated Tool
 
 ## Executing tests
 
-Once setup is complete, you may run unit/integration tests using the following :
-
-```
-## Run a unit test
-make test
-
-## Run integration tests
-make it
-```
-
-### Important information related to integration tests
+Once setup is complete, you may run the unit and integration tests.
 
-To have all the tests run successfully JDK 17 is required, since one of the tests builds the latest version of OpenSearch from source, and the OpenSearch project uses JDK 17 for this purpose.
-```
-export JAVA_HOME=/path/to/JDK17
+### Unit Tests
 
 ```
+make test
 
-Note that the `javadoc` executable should be available in the JDK installation.
-
-You could also use one of the following versions instead: 16, 15, 14, 13, 12, 11 or 8.  Most of the complement of tests included will work with these.
-
-If you have multiple JDKs installed, export them in the following format `JAVA(jdk_version)_HOME`. Here is an example of how one would export JDK 8, 11, 15, 16:
-```
-export JAVA8_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_231.jdk/Contents/Home/
-export JAVA11_HOME=/Library/Java/JavaVirtualMachines/jdk-11.0.8.jdk/Contents/Home
-export JAVA15_HOME=/Library/Java/JavaVirtualMachines/amazon-corretto-15.jdk/Contents/Home/
-export JAVA16_HOME=/Library/Java/JavaVirtualMachines/amazon-corretto-16.jdk/Contents/Home/
 ```
 
-OpenSearch currently releases artifacts for the following operating systems:
-- Linux
-- Docker
-- FreeBSD
+### Integration Tests
 
-If the operating system is not listed above then the artifacts used will default to Linux.
+Integration tests can be run on the following operating systems:
+  * RedHat
+  * CentOS
+  * Ubuntu
+  * Amazon Linux 2
+  * MacOS
 
-For MacOS users running OpenSearch, please set JAVA_HOME to one of the local JDKs you exported as the JDK bundled with OpenSearch is not compatible with MacOS and prevent the metrics store from coming up correctly during integration tests.
 
-### Troubleshooting
-```
-root WARNING Unable to get address info for address xxxxyyyy (AddressFamily.AF_INET, SocketKind.SOCK_DGRAM, 17, 0): <class 'socket.gaierror'> [Errno 8] nodename nor servname provided, or not known
 ```
-- VPN may be interfering. Try turning off the VPN. If you are connected to a VPN and face Docker-related issues when running the integration tests disconnecting from the VPN may fix this.
+make it
 
 ```
-Cannot download OpenSearch-1.0.1
-```
-- JAVA_HOME may not be correctly set.
-
 
 ## Submitting your changes for a pull request
 

Makefile

@@ -49,6 +49,15 @@ prereq:
 	@ jq -r '.python_versions | [.[] | tostring] | join("\n")' .ci/variables.json > .python-version
 	-@ printf $(PYENV_PREREQ_HELP)
 
+check-java:
+	@if ! test "$(JAVA_HOME)" || ! java --version > /dev/null 2>&1 || ! javadoc --help > /dev/null 2>&1; then \
+	    echo "Java installation issues for running integration tests" >&2; \
+	    exit 1; \
+	fi
+	@if test `java --version | sed -n 's/[^0-9]*\([0-9]*\).*./\1/p;q'` != 17; then \
+	    echo "NOTE: Java version 17 required to have all integration tests pass" >&2; \
+	fi
+
 venv-create:
 	@if [[ ! -x $$(command -v pyenv) ]]; then \
 		printf $(PYENV_ERROR); \
@@ -112,20 +121,11 @@ test: check-venv
 
 precommit: lint
 
-it: check-venv python-caches-clean tox-env-clean
+it: check-java check-venv python-caches-clean tox-env-clean
 	. $(VENV_ACTIVATE_FILE); tox
 
-it38: check-venv python-caches-clean tox-env-clean
-	. $(VENV_ACTIVATE_FILE); tox -e py38
-
-it39: check-venv python-caches-clean tox-env-clean
-	. $(VENV_ACTIVATE_FILE); tox -e py39
-
-it310: check-venv python-caches-clean tox-env-clean
-	. $(VENV_ACTIVATE_FILE); tox -e py310
-
-it311: check-venv python-caches-clean tox-env-clean
-	. $(VENV_ACTIVATE_FILE); tox -e py311
+it38 it39 it310 it311: check-java check-venv python-caches-clean tox-env-clean
+	. $(VENV_ACTIVATE_FILE); tox -e $(@:it%=py%)
 
 benchmark: check-venv
 	. $(VENV_ACTIVATE_FILE); pytest benchmarks/

it/__init__.py

@@ -86,7 +86,10 @@ def osbenchmark(cfg, command_line):
     This method should be used for benchmark invocations of the all commands besides test_execution.
     These commands may have different CLI options than test_execution.
     """
-    return os.system(osbenchmark_command_line_for(cfg, command_line))
+    err, retcode = process.run_subprocess_get_stderr(osbenchmark_command_line_for(cfg, command_line))
+    if retcode != 0:
+        print(err)
+    return retcode
 
 
 def execute_test(cfg, command_line):
@@ -163,8 +166,8 @@ def install(self, distribution_version, node_name, provision_config_instance, ht
         self.http_port = http_port
         transport_port = http_port + 100
         try:
-            output = process.run_subprocess_with_output(
-                "opensearch-benchmark install --configuration-name={cfg} --quiet --distribution-version={dist} --build-type=tar "
+            err, retcode = process.run_subprocess_get_stderr(
+                "opensearch-benchmark install --configuration-name={cfg} --distribution-version={dist} --build-type=tar "
                 "--http-port={http_port} --node={node_name} --master-nodes="
                 "{node_name} --provision-config-instance={provision_config_instance} "
                 "--seed-hosts=\"127.0.0.1:{transport_port}\"".format(cfg=self.cfg,
@@ -173,7 +176,9 @@ def install(self, distribution_version, node_name, provision_config_instance, ht
                                                                      node_name=node_name,
                                                                      provision_config_instance=provision_config_instance,
                                                                      transport_port=transport_port))
-            self.installation_id = json.loads("".join(output))["installation-id"]
+            if retcode != 0:
+                raise AssertionError("Failed to install OpenSearch {}.".format(distribution_version), err)
+            self.installation_id = json.loads(err)["installation-id"]
         except BaseException as e:
             raise AssertionError("Failed to install OpenSearch {}.".format(distribution_version), e)
 

osbenchmark/builder/builder.py

@@ -81,7 +81,7 @@ def install(cfg):
         raise exceptions.SystemSetupError("Unknown build type [{}]".format(build_type))
 
     provisioner.save_node_configuration(root_path, node_config)
-    console.println(json.dumps({"installation-id": cfg.opts("system", "install.id")}, indent=2), force=True)
+    console.println(json.dumps({"installation-id": cfg.opts("system", "install.id")}, indent=2), force=True, stderr=True)
 
 
 def start(cfg):

osbenchmark/resources/provision_configs/1.0/provision_config_instances/v1/vanilla/config.ini

@@ -16,7 +16,7 @@ jdk.unbundled.release_url = https://artifacts.opensearch.org/releases/bundle/ope
 
 docker_image=opensearchproject/opensearch
 # major version of the JDK that is used to build OpenSearch
-build.jdk = 12
+build.jdk = 17
 # list of JDK major versions that are used to run OpenSearch
 runtime.jdk = 17,16,15,14,13,12,11,8
 runtime.jdk.bundled = true

osbenchmark/utils/console.py

@@ -150,18 +150,19 @@ def warn(msg, end="\n", flush=False, force=False, logger=None, overline=None, un
 
 def error(msg, end="\n", flush=False, force=False, logger=None, overline=None, underline=None):
     println(msg, console_prefix="[ERROR]", end=end, flush=flush, force=force, overline=overline, underline=underline
-            , logger=logger.error if logger else None)
+            , logger=logger.error if logger else None, stderr=True)
 
 
-def println(msg, console_prefix=None, end="\n", flush=False, force=False, logger=None, overline=None, underline=None):
+def println(msg, console_prefix=None, end="\n", flush=False, force=False, logger=None, overline=None, underline=None, stderr=False):
     allow_print = force or (not QUIET and (BENCHMARK_RUNNING_IN_DOCKER or ASSUME_TTY or sys.stdout.isatty()))
+    file=sys.stderr if stderr else sys.stdout
     if allow_print:
         complete_msg = "%s %s" % (console_prefix, msg) if console_prefix else msg
         if overline:
-            print(format.underline_for(complete_msg, underline_symbol=overline), flush=flush)
-        print(complete_msg, end=end, flush=flush)
+            print(format.underline_for(complete_msg, underline_symbol=overline), flush=flush, file=file)
+        print(complete_msg, end=end, flush=flush, file=file)
         if underline:
-            print(format.underline_for(complete_msg, underline_symbol=underline), flush=flush)
+            print(format.underline_for(complete_msg, underline_symbol=underline), flush=flush, file=file)
     if logger:
         logger(msg)
 

osbenchmark/utils/process.py

@@ -52,6 +52,17 @@ def run_subprocess_with_output(command_line):
     return lines
 
 
+def run_subprocess_get_stderr(command_line):
+    logger = logging.getLogger(__name__)
+    logger.debug("Running subprocess [%s] with output and err.", command_line)
+    command_line_args = shlex.split(command_line)
+
+    sp = subprocess.Popen(command_line_args, stdout=subprocess.DEVNULL, stderr=subprocess.PIPE)
+    sp.wait()
+    _, err = sp.communicate()
+    return err.decode("UTF-8"), sp.returncode
+
+
 def exit_status_as_bool(runnable, quiet=False):
     """
 
@@ -110,10 +121,12 @@ def run_subprocess_with_logging(command_line, header=None, level=logging.INFO, s
 
 
 def is_benchmark_process(p):
+    cmdline = p.cmdline()
     return p.name() == "opensearch-benchmark" or \
-           (p.name().lower().startswith("python")
-            and any("opensearch-benchmark" in e for e in p.cmdline())
-            and not any("opensearch-benchmarkd" in e for e in p.cmdline()))
+           (p.name().lower().startswith("python") and
+            (len(cmdline) > 1 and
+             (cmdline[1] == "opensearch-benchmark" or
+              cmdline[1].endswith(os.path.sep + "opensearch-benchmark"))))
 
 
 def find_all_other_benchmark_processes():

tests/utils/console_test.py

@@ -23,6 +23,7 @@
 # under the License.
 
 import os
+import sys
 import random
 import unittest.mock as mock
 from unittest import TestCase
@@ -74,7 +75,7 @@ def test_println_randomized_dockertrue_or_istty_and_isnotquiet(self, patched_pri
 
         console.println(msg="Unittest message")
         patched_print.assert_called_once_with(
-            "Unittest message", end="\n", flush=False
+            "Unittest message", end="\n", flush=False, file=sys.stdout
         )
 
     @mock.patch("sys.stdout.isatty")
@@ -85,7 +86,7 @@ def test_println_randomized_assume_tty_or_istty_and_isnotquiet(self, patched_pri
         patched_isatty.return_value = random_boolean
         console.println(msg="Unittest message")
         patched_print.assert_called_once_with(
-            "Unittest message", end="\n", flush=False
+            "Unittest message", end="\n", flush=False, file=sys.stdout
         )
 
     @mock.patch("sys.stdout.isatty")
@@ -106,7 +107,7 @@ def test_println_force_prints_even_when_quiet(self, patched_print, patched_isatt
 
         console.println(msg="Unittest message", force=True)
         patched_print.assert_called_once_with(
-            "Unittest message", end="\n", flush=False
+            "Unittest message", end="\n", flush=False, file=sys.stdout
         )