Skip to content

Commit 51fe203

Browse files
authored
More fixes for the integration tests (opensearch-project#317)
Signed-off-by: Govind Kamat <govkamat@amazon.com>
1 parent f5370c3 commit 51fe203

File tree

9 files changed

+79
-83
lines changed

9 files changed

+79
-83
lines changed

.ci/build.sh

+2-9
Original file line numberDiff line numberDiff line change
@@ -85,15 +85,8 @@ function build_it {
8585
make install
8686
make precommit
8787

88-
if [[ "$1" == "3.8" ]]; then
89-
make it38
90-
elif [[ "$1" == "3.9" ]]; then
91-
make it39
92-
elif [[ "$1" == "3.10" ]]; then
93-
make it310
94-
elif [[ "$1" == "3.11" ]]; then
95-
make it311
96-
fi
88+
# make it38, it39, etc.
89+
make "it${1//./}"
9790
}
9891

9992
function license-scan {

DEVELOPER_GUIDE.md

+28-45
Original file line numberDiff line numberDiff line change
@@ -3,19 +3,29 @@
33
This document will walk you through on what's needed to start contributing code to OpenSearch Benchmark.
44

55
## Installation
6+
67
### Prerequisites
78

8-
- Pyenv : Install `pyenv` and follow the instructions in the output of `pyenv init` to setup your shell and restart it before proceeding.
9-
For more details please refer to the [PyEnv installation instructions](https://github.com/pyenv/pyenv#installation).
10-
- JDK : JDK version required to build OpenSearch. Please refer to the [build setup requirements](https://github.com/opensearch-project/OpenSearch/blob/ca564fd04f5059cf9e3ce8aba442575afb3d99f1/DEVELOPER_GUIDE.md#install-prerequisites).
11-
- Docker : Docker and additionally `docker-compose` on Linux.
12-
- Git : git 1.9 or latter.
9+
- Pyenv : Install `pyenv` and follow the instructions in the output of `pyenv init` to set up your shell and restart it before proceeding.
10+
For more details please refer to the [PyEnv installation instructions](https://github.com/pyenv/pyenv#installation).
11+
12+
- JDK: Although OSB is a Python application, it optionally builds and provisions OpenSearch clusters. JDK version 17 is used to build the current version of OpenSearch. Please refer to the [build setup requirements](https://github.com/opensearch-project/OpenSearch/blob/ca564fd04f5059cf9e3ce8aba442575afb3d99f1/DEVELOPER_GUIDE.md#install-prerequisites).
13+
Note that the `javadoc` executable should be available in the JDK installation. An earlier version of the JDK can be used, but not all the integration tests will pass.
14+
15+
```
16+
export JAVA_HOME=/path/to/JDK17
17+
18+
```
19+
20+
- Install Docker and `docker-compose`. Start the Docker server. The user running the integration tests should have the permissions required to run docker commands. Test by running `docker ps`.
21+
22+
- Git : git 1.9 or later.
1323
1424
### Setup
1525
16-
Use the following command-line instructions to setup OpenSearch Benchmark for development:
26+
Use the following command-line instructions to set up OpenSearch Benchmark for development:
1727
```
18-
git clone https://github.com/opensearch-project/OpenSearch-Benchmark.git
28+
git clone https://github.com/opensearch-project/OpenSearch-Benchmark
1929
cd OpenSearch-Benchmark
2030
make prereq
2131
make install
@@ -52,56 +62,29 @@ In order to run tests within the PyCharm IDE, ensure the `Python Integrated Tool
5262
5363
## Executing tests
5464
55-
Once setup is complete, you may run unit/integration tests using the following :
56-
57-
```
58-
## Run a unit test
59-
make test
60-
61-
## Run integration tests
62-
make it
63-
```
64-
65-
### Important information related to integration tests
65+
Once setup is complete, you may run the unit and integration tests.
6666
67-
To have all the tests run successfully JDK 17 is required, since one of the tests builds the latest version of OpenSearch from source, and the OpenSearch project uses JDK 17 for this purpose.
68-
```
69-
export JAVA_HOME=/path/to/JDK17
67+
### Unit Tests
7068
7169
```
70+
make test
7271

73-
Note that the `javadoc` executable should be available in the JDK installation.
74-
75-
You could also use one of the following versions instead: 16, 15, 14, 13, 12, 11 or 8. Most of the complement of tests included will work with these.
76-
77-
If you have multiple JDKs installed, export them in the following format `JAVA(jdk_version)_HOME`. Here is an example of how one would export JDK 8, 11, 15, 16:
78-
```
79-
export JAVA8_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_231.jdk/Contents/Home/
80-
export JAVA11_HOME=/Library/Java/JavaVirtualMachines/jdk-11.0.8.jdk/Contents/Home
81-
export JAVA15_HOME=/Library/Java/JavaVirtualMachines/amazon-corretto-15.jdk/Contents/Home/
82-
export JAVA16_HOME=/Library/Java/JavaVirtualMachines/amazon-corretto-16.jdk/Contents/Home/
8372
```
8473
85-
OpenSearch currently releases artifacts for the following operating systems:
86-
- Linux
87-
- Docker
88-
- FreeBSD
74+
### Integration Tests
8975
90-
If the operating system is not listed above then the artifacts used will default to Linux.
76+
Integration tests can be run on the following operating systems:
77+
* RedHat
78+
* CentOS
79+
* Ubuntu
80+
* Amazon Linux 2
81+
* MacOS
9182
92-
For MacOS users running OpenSearch, please set JAVA_HOME to one of the local JDKs you exported as the JDK bundled with OpenSearch is not compatible with MacOS and prevent the metrics store from coming up correctly during integration tests.
9383
94-
### Troubleshooting
95-
```
96-
root WARNING Unable to get address info for address xxxxyyyy (AddressFamily.AF_INET, SocketKind.SOCK_DGRAM, 17, 0): <class 'socket.gaierror'> [Errno 8] nodename nor servname provided, or not known
9784
```
98-
- VPN may be interfering. Try turning off the VPN. If you are connected to a VPN and face Docker-related issues when running the integration tests disconnecting from the VPN may fix this.
85+
make it
9986

10087
```
101-
Cannot download OpenSearch-1.0.1
102-
```
103-
- JAVA_HOME may not be correctly set.
104-
10588
10689
## Submitting your changes for a pull request
10790

Makefile

+12-12
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,15 @@ prereq:
4949
@ jq -r '.python_versions | [.[] | tostring] | join("\n")' .ci/variables.json > .python-version
5050
-@ printf $(PYENV_PREREQ_HELP)
5151

52+
check-java:
53+
@if ! test "$(JAVA_HOME)" || ! java --version > /dev/null 2>&1 || ! javadoc --help > /dev/null 2>&1; then \
54+
echo "Java installation issues for running integration tests" >&2; \
55+
exit 1; \
56+
fi
57+
@if test `java --version | sed -n 's/[^0-9]*\([0-9]*\).*./\1/p;q'` != 17; then \
58+
echo "NOTE: Java version 17 required to have all integration tests pass" >&2; \
59+
fi
60+
5261
venv-create:
5362
@if [[ ! -x $$(command -v pyenv) ]]; then \
5463
printf $(PYENV_ERROR); \
@@ -112,20 +121,11 @@ test: check-venv
112121

113122
precommit: lint
114123

115-
it: check-venv python-caches-clean tox-env-clean
124+
it: check-java check-venv python-caches-clean tox-env-clean
116125
. $(VENV_ACTIVATE_FILE); tox
117126

118-
it38: check-venv python-caches-clean tox-env-clean
119-
. $(VENV_ACTIVATE_FILE); tox -e py38
120-
121-
it39: check-venv python-caches-clean tox-env-clean
122-
. $(VENV_ACTIVATE_FILE); tox -e py39
123-
124-
it310: check-venv python-caches-clean tox-env-clean
125-
. $(VENV_ACTIVATE_FILE); tox -e py310
126-
127-
it311: check-venv python-caches-clean tox-env-clean
128-
. $(VENV_ACTIVATE_FILE); tox -e py311
127+
it38 it39 it310 it311: check-java check-venv python-caches-clean tox-env-clean
128+
. $(VENV_ACTIVATE_FILE); tox -e $(@:it%=py%)
129129

130130
benchmark: check-venv
131131
. $(VENV_ACTIVATE_FILE); pytest benchmarks/

it/__init__.py

+9-4
Original file line numberDiff line numberDiff line change
@@ -86,7 +86,10 @@ def osbenchmark(cfg, command_line):
8686
This method should be used for benchmark invocations of the all commands besides test_execution.
8787
These commands may have different CLI options than test_execution.
8888
"""
89-
return os.system(osbenchmark_command_line_for(cfg, command_line))
89+
err, retcode = process.run_subprocess_get_stderr(osbenchmark_command_line_for(cfg, command_line))
90+
if retcode != 0:
91+
print(err)
92+
return retcode
9093

9194

9295
def execute_test(cfg, command_line):
@@ -163,8 +166,8 @@ def install(self, distribution_version, node_name, provision_config_instance, ht
163166
self.http_port = http_port
164167
transport_port = http_port + 100
165168
try:
166-
output = process.run_subprocess_with_output(
167-
"opensearch-benchmark install --configuration-name={cfg} --quiet --distribution-version={dist} --build-type=tar "
169+
err, retcode = process.run_subprocess_get_stderr(
170+
"opensearch-benchmark install --configuration-name={cfg} --distribution-version={dist} --build-type=tar "
168171
"--http-port={http_port} --node={node_name} --master-nodes="
169172
"{node_name} --provision-config-instance={provision_config_instance} "
170173
"--seed-hosts=\"127.0.0.1:{transport_port}\"".format(cfg=self.cfg,
@@ -173,7 +176,9 @@ def install(self, distribution_version, node_name, provision_config_instance, ht
173176
node_name=node_name,
174177
provision_config_instance=provision_config_instance,
175178
transport_port=transport_port))
176-
self.installation_id = json.loads("".join(output))["installation-id"]
179+
if retcode != 0:
180+
raise AssertionError("Failed to install OpenSearch {}.".format(distribution_version), err)
181+
self.installation_id = json.loads(err)["installation-id"]
177182
except BaseException as e:
178183
raise AssertionError("Failed to install OpenSearch {}.".format(distribution_version), e)
179184

osbenchmark/builder/builder.py

+1-1
Original file line numberDiff line numberDiff line change
@@ -81,7 +81,7 @@ def install(cfg):
8181
raise exceptions.SystemSetupError("Unknown build type [{}]".format(build_type))
8282

8383
provisioner.save_node_configuration(root_path, node_config)
84-
console.println(json.dumps({"installation-id": cfg.opts("system", "install.id")}, indent=2), force=True)
84+
console.println(json.dumps({"installation-id": cfg.opts("system", "install.id")}, indent=2), force=True, stderr=True)
8585

8686

8787
def start(cfg):

osbenchmark/resources/provision_configs/1.0/provision_config_instances/v1/vanilla/config.ini

+1-1
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ jdk.unbundled.release_url = https://artifacts.opensearch.org/releases/bundle/ope
1616

1717
docker_image=opensearchproject/opensearch
1818
# major version of the JDK that is used to build OpenSearch
19-
build.jdk = 12
19+
build.jdk = 17
2020
# list of JDK major versions that are used to run OpenSearch
2121
runtime.jdk = 17,16,15,14,13,12,11,8
2222
runtime.jdk.bundled = true

osbenchmark/utils/console.py

+6-5
Original file line numberDiff line numberDiff line change
@@ -150,18 +150,19 @@ def warn(msg, end="\n", flush=False, force=False, logger=None, overline=None, un
150150

151151
def error(msg, end="\n", flush=False, force=False, logger=None, overline=None, underline=None):
152152
println(msg, console_prefix="[ERROR]", end=end, flush=flush, force=force, overline=overline, underline=underline
153-
, logger=logger.error if logger else None)
153+
, logger=logger.error if logger else None, stderr=True)
154154

155155

156-
def println(msg, console_prefix=None, end="\n", flush=False, force=False, logger=None, overline=None, underline=None):
156+
def println(msg, console_prefix=None, end="\n", flush=False, force=False, logger=None, overline=None, underline=None, stderr=False):
157157
allow_print = force or (not QUIET and (BENCHMARK_RUNNING_IN_DOCKER or ASSUME_TTY or sys.stdout.isatty()))
158+
file=sys.stderr if stderr else sys.stdout
158159
if allow_print:
159160
complete_msg = "%s %s" % (console_prefix, msg) if console_prefix else msg
160161
if overline:
161-
print(format.underline_for(complete_msg, underline_symbol=overline), flush=flush)
162-
print(complete_msg, end=end, flush=flush)
162+
print(format.underline_for(complete_msg, underline_symbol=overline), flush=flush, file=file)
163+
print(complete_msg, end=end, flush=flush, file=file)
163164
if underline:
164-
print(format.underline_for(complete_msg, underline_symbol=underline), flush=flush)
165+
print(format.underline_for(complete_msg, underline_symbol=underline), flush=flush, file=file)
165166
if logger:
166167
logger(msg)
167168

osbenchmark/utils/process.py

+16-3
Original file line numberDiff line numberDiff line change
@@ -52,6 +52,17 @@ def run_subprocess_with_output(command_line):
5252
return lines
5353

5454

55+
def run_subprocess_get_stderr(command_line):
56+
logger = logging.getLogger(__name__)
57+
logger.debug("Running subprocess [%s] with output and err.", command_line)
58+
command_line_args = shlex.split(command_line)
59+
60+
sp = subprocess.Popen(command_line_args, stdout=subprocess.DEVNULL, stderr=subprocess.PIPE)
61+
sp.wait()
62+
_, err = sp.communicate()
63+
return err.decode("UTF-8"), sp.returncode
64+
65+
5566
def exit_status_as_bool(runnable, quiet=False):
5667
"""
5768
@@ -110,10 +121,12 @@ def run_subprocess_with_logging(command_line, header=None, level=logging.INFO, s
110121

111122

112123
def is_benchmark_process(p):
124+
cmdline = p.cmdline()
113125
return p.name() == "opensearch-benchmark" or \
114-
(p.name().lower().startswith("python")
115-
and any("opensearch-benchmark" in e for e in p.cmdline())
116-
and not any("opensearch-benchmarkd" in e for e in p.cmdline()))
126+
(p.name().lower().startswith("python") and
127+
(len(cmdline) > 1 and
128+
(cmdline[1] == "opensearch-benchmark" or
129+
cmdline[1].endswith(os.path.sep + "opensearch-benchmark"))))
117130

118131

119132
def find_all_other_benchmark_processes():

tests/utils/console_test.py

+4-3
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@
2323
# under the License.
2424

2525
import os
26+
import sys
2627
import random
2728
import unittest.mock as mock
2829
from unittest import TestCase
@@ -74,7 +75,7 @@ def test_println_randomized_dockertrue_or_istty_and_isnotquiet(self, patched_pri
7475

7576
console.println(msg="Unittest message")
7677
patched_print.assert_called_once_with(
77-
"Unittest message", end="\n", flush=False
78+
"Unittest message", end="\n", flush=False, file=sys.stdout
7879
)
7980

8081
@mock.patch("sys.stdout.isatty")
@@ -85,7 +86,7 @@ def test_println_randomized_assume_tty_or_istty_and_isnotquiet(self, patched_pri
8586
patched_isatty.return_value = random_boolean
8687
console.println(msg="Unittest message")
8788
patched_print.assert_called_once_with(
88-
"Unittest message", end="\n", flush=False
89+
"Unittest message", end="\n", flush=False, file=sys.stdout
8990
)
9091

9192
@mock.patch("sys.stdout.isatty")
@@ -106,7 +107,7 @@ def test_println_force_prints_even_when_quiet(self, patched_print, patched_isatt
106107

107108
console.println(msg="Unittest message", force=True)
108109
patched_print.assert_called_once_with(
109-
"Unittest message", end="\n", flush=False
110+
"Unittest message", end="\n", flush=False, file=sys.stdout
110111
)
111112

112113

0 commit comments

Comments
 (0)