Skip to content

Commit 758dd21

Browse files
derek-hoLDrago27
derek-ho
authored and
LDrago27
committed
Add infra support and helper function to create real datasource (opensearch-project#1146)
* Add infra support and helper function to create real datasource Signed-off-by: Derek Ho <dxho@amazon.com> * Remove logs from previous change Signed-off-by: Derek Ho <dxho@amazon.com> * Add test that uses the helper function to verify functionality Signed-off-by: Derek Ho <dxho@amazon.com> * Change to request and add it to appropriate workflows Signed-off-by: Derek Ho <dxho@amazon.com> * Add basic auth remote cluster Signed-off-by: Derek Ho <dxho@amazon.com> * Call function Signed-off-by: Derek Ho <dxho@amazon.com> * Move action Signed-off-by: Derek Ho <dxho@amazon.com> * Update action and usage in docs Signed-off-by: Derek Ho <dxho@amazon.com> * Checkout before access Signed-off-by: Derek Ho <dxho@amazon.com> * Add relative path Signed-off-by: Derek Ho <dxho@amazon.com> * Fix spelling Signed-off-by: Derek Ho <dxho@amazon.com> * Remove -d Signed-off-by: Derek Ho <dxho@amazon.com> * Move port logic Signed-off-by: Derek Ho <dxho@amazon.com> * Cat log Signed-off-by: Derek Ho <dxho@amazon.com> * Allow modification Signed-off-by: Derek Ho <dxho@amazon.com> * Duplicate download Signed-off-by: Derek Ho <dxho@amazon.com> * Change directory Signed-off-by: Derek Ho <dxho@amazon.com> * Change name Signed-off-by: Derek Ho <dxho@amazon.com> * Remove extra chmod Signed-off-by: Derek Ho <dxho@amazon.com> * Within test directory Signed-off-by: Derek Ho <dxho@amazon.com> * Fix Signed-off-by: Derek Ho <dxho@amazon.com> * Fix for windows, add helper function, show usage Signed-off-by: Derek Ho <dxho@amazon.com> * Remove pwd Signed-off-by: Derek Ho <dxho@amazon.com> * Move to temp dire Signed-off-by: Derek Ho <dxho@amazon.com> * Address PR feedback and try to spin up an instance with security Signed-off-by: Derek Ho <dxho@amazon.com> * Small action change and documentation add Signed-off-by: Derek Ho <dxho@amazon.com> --------- Signed-off-by: Derek Ho <dxho@amazon.com>
1 parent 918840b commit 758dd21

File tree

8 files changed

+330
-0
lines changed

8 files changed

+330
-0
lines changed

‎.github/actions/start-opensearch/action.yml

+195
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,195 @@
1+
name: 'Launch OpenSearch'
2+
description: 'Downloads OpenSearch and runs it'
3+
4+
inputs:
5+
opensearch-version:
6+
description: 'The version of OpenSearch that should be used, e.g "3.0.0"'
7+
required: true
8+
9+
plugins:
10+
description: 'A comma separated list of plugins to install. Leave empty to not install any. Each entry should be a full path prefixed with `file: `, for example: `file:$(pwd)/my-plugin.zip`'
11+
required: false
12+
13+
security-enabled:
14+
description: 'Whether security is enabled'
15+
required: true
16+
17+
admin-password:
18+
description: 'The admin password uses for the cluster'
19+
required: false
20+
21+
security_config_file:
22+
description: 'Path to a security config file to replace the default. Leave empty if security is not enabled or using the default config'
23+
required: false
24+
25+
port:
26+
description: 'Port to run OpenSearch. Leave empty to use the default config (9200)'
27+
required: false
28+
default: '9200'
29+
30+
runs:
31+
using: "composite"
32+
steps:
33+
34+
# Configure longpath names if on Windows
35+
- name: Enable Longpaths if on Windows
36+
if: ${{ runner.os == 'Windows' }}
37+
run: git config --system core.longpaths true
38+
shell: pwsh
39+
40+
- name: Set up JDK
41+
uses: actions/setup-java@v1
42+
with:
43+
java-version: 11
44+
45+
- name: Create temporary directory
46+
run: |
47+
mkdir ${{ inputs.port }}
48+
shell: bash
49+
50+
# Download OpenSearch
51+
- name: Download OpenSearch for Windows
52+
uses: peternied/download-file@v2
53+
if: ${{ runner.os == 'Windows' }}
54+
with:
55+
url: https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/${{ inputs.opensearch-version }}/latest/windows/x64/zip/dist/opensearch/opensearch-${{ inputs.opensearch-version }}-windows-x64.zip
56+
57+
58+
- name: Download OpenSearch for Linux
59+
uses: peternied/download-file@v2
60+
if: ${{ runner.os == 'Linux' }}
61+
with:
62+
url: https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/${{ inputs.opensearch-version }}/latest/linux/x64/tar/dist/opensearch/opensearch-${{ inputs.opensearch-version }}-linux-x64.tar.gz
63+
64+
# Extract downloaded zip
65+
- name: Extract downloaded tar
66+
if: ${{ runner.os == 'Linux' }}
67+
run: |
68+
tar -xzf opensearch-*.tar.gz -C ${{ inputs.port }} && mv ${{ inputs.port }}/opensearch-${{ inputs.opensearch-version }} ${{ inputs.port }}/opensearch-${{inputs.opensearch-version}}-SNAPSHOT${{ inputs.port }}
69+
rm -f opensearch-*.tar.gz
70+
shell: bash
71+
72+
- name: Extract downloaded zip
73+
if: ${{ runner.os == 'Windows' }}
74+
run: |
75+
Expand-Archive -Path "opensearch-${{ inputs.opensearch-version }}-windows-x64.zip" -DestinationPath "temp"
76+
Move-Item -Path "temp/*" -Destination "${{ inputs.port }}/opensearch-${{ inputs.opensearch-version }}-SNAPSHOT${{ inputs.port }}"
77+
Remove-Item -Path "temp" -Recurse
78+
Remove-Item -Path "opensearch-${{ inputs.opensearch-version }}-windows-x64.zip.1"
79+
shell: pwsh
80+
81+
# Setup security if it's enabled
82+
- name: Setup security demo configuration
83+
if: ${{ runner.os == 'Linux' && inputs.security-enabled == 'true' }}
84+
run: |
85+
echo "running linux security demo configuration setup"
86+
export OPENSEARCH_INITIAL_ADMIN_PASSWORD=${{ inputs.admin-password }}
87+
chmod +x ./opensearch-${{ inputs.opensearch-version }}-SNAPSHOT${{ inputs.port }}/plugins/opensearch-security/tools/install_demo_configuration.sh
88+
opensearch_version="${{ inputs.opensearch-version }}"
89+
opensearch_major_version=$(echo "$opensearch_version" | awk -F'.' '{print $1}')
90+
opensearch_minor_version=$(echo "$opensearch_version" | awk -F'.' '{print $2}')
91+
if [ "$opensearch_major_version" -lt 2 ] || ([ "$opensearch_major_version" -eq 2 ] && [ "$opensearch_minor_version" -lt 12 ]); then
92+
echo "Running the command without -t option (OpenSearch version is $opensearch_version)"
93+
/bin/bash -c "yes | ./opensearch-${opensearch_version}-SNAPSHOT${{ inputs.port }}/plugins/opensearch-security/tools/install_demo_configuration.sh"
94+
else
95+
echo "Running the command with -t option (OpenSearch version is $opensearch_version)"
96+
/bin/bash -c "yes | ./opensearch-${opensearch_version}-SNAPSHOT${{ inputs.port }}/plugins/opensearch-security/tools/install_demo_configuration.sh -t"
97+
fi
98+
echo "plugins.security.unsupported.restapi.allow_securityconfig_modification: true" >> ./opensearch-${{ inputs.opensearch-version }}-SNAPSHOT${{ inputs.port }}/config/opensearch.yml
99+
shell: bash
100+
working-directory: ${{ inputs.port }}
101+
102+
- name: Setup security demo configuration for Windows
103+
if: ${{ runner.os == 'Windows' && inputs.security-enabled == 'true' }}
104+
run: |
105+
echo "running windows security demo configuration setup"
106+
export OPENSEARCH_INITIAL_ADMIN_PASSWORD=${{ inputs.admin-password }}
107+
chmod +x ./opensearch-${{ inputs.opensearch-version }}-SNAPSHOT${{ inputs.port }}/plugins/opensearch-security/tools/install_demo_configuration.bat
108+
opensearch_version="${{ inputs.opensearch-version }}"
109+
opensearch_major_version=$(echo "$opensearch_version" | cut -d'.' -f1)
110+
opensearch_minor_version=$(echo "$opensearch_version" | cut -d'.' -f2)
111+
if [ "$opensearch_major_version" -lt 2 ] || ([ "$opensearch_major_version" -eq 2 ] && [ "$opensearch_minor_version" -lt 12 ]); then
112+
echo "Running the command without -t option (OpenSearch version is $opensearch_version)"
113+
/bin/bash -c "yes | ./opensearch-${opensearch_version}-SNAPSHOT${{ inputs.port }}/plugins/opensearch-security/tools/install_demo_configuration.bat -y -i -s"
114+
else
115+
echo "Running the command with -t option (OpenSearch version is $opensearch_version)"
116+
/bin/bash -c "yes | ./opensearch-${opensearch_version}-SNAPSHOT${{ inputs.port }}/plugins/opensearch-security/tools/install_demo_configuration.bat -t -y -i -s"
117+
fi
118+
echo "plugins.security.unsupported.restapi.allow_securityconfig_modification: true" >> ./opensearch-${{ inputs.opensearch-version }}-SNAPSHOT${{ inputs.port }}/config/opensearch.yml
119+
shell: bash
120+
working-directory: ${{ inputs.port }}
121+
122+
# Disable security if it's disabled
123+
- name: Disable security
124+
if: ${{ inputs.security-enabled == 'false' }}
125+
run: |
126+
echo "Remove OpenSearch Security"
127+
echo "plugins.security.disabled: true" >> ./opensearch-${{inputs.opensearch-version}}-SNAPSHOT${{ inputs.port }}/config/opensearch.yml
128+
shell: bash
129+
working-directory: ${{ inputs.port }}
130+
131+
- name: Use more space
132+
run: |
133+
echo '' >> ./opensearch-${{ inputs.opensearch-version }}-SNAPSHOT${{ inputs.port }}/config/opensearch.yml
134+
echo "cluster.routing.allocation.disk.threshold_enabled: false" >> ./opensearch-${{ inputs.opensearch-version }}-SNAPSHOT${{ inputs.port }}/config/opensearch.yml
135+
echo "http.port: ${{ inputs.port }}" >> ./opensearch-${{ inputs.opensearch-version }}-SNAPSHOT${{ inputs.port }}/config/opensearch.yml
136+
shell: bash
137+
working-directory: ${{ inputs.port }}
138+
139+
# Run OpenSearch
140+
- name: Run OpenSearch with plugin on Linux
141+
if: ${{ runner.os == 'Linux'}}
142+
run: /bin/bash -c "./opensearch-${{ inputs.opensearch-version }}-SNAPSHOT${{ inputs.port }}/bin/opensearch &"
143+
shell: bash
144+
working-directory: ${{ inputs.port }}
145+
146+
- name: Run OpenSearch with plugin on Windows
147+
if: ${{ runner.os == 'Windows'}}
148+
run: start .\opensearch-${{ inputs.opensearch-version }}-SNAPSHOT${{ inputs.port }}\bin\opensearch.bat
149+
shell: pwsh
150+
working-directory: ${{ inputs.port }}
151+
152+
# Give the OpenSearch process some time to boot up before sending any requires, might need to increase the default time!
153+
- name: Sleep while OpenSearch starts
154+
uses: peternied/action-sleep@v1
155+
with:
156+
seconds: 30
157+
158+
# Verify that the server is operational
159+
- name: Check OpenSearch Running on Linux
160+
if: ${{ runner.os != 'Windows'}}
161+
run: |
162+
if [ "${{ inputs.security-enabled }}" == "true" ]; then
163+
curl https://localhost:${{ inputs.port || 9200 }}/_cat/plugins -u 'admin:${{ inputs.admin-password }}' -k -v --fail-with-body
164+
else
165+
curl http://localhost:${{ inputs.port || 9200 }}/_cat/plugins -v
166+
fi
167+
shell: bash
168+
working-directory: ${{ inputs.port }}
169+
170+
- name: Check OpenSearch Running on Windows
171+
if: ${{ runner.os == 'Windows'}}
172+
run: |
173+
if ("${{ inputs.security-enabled }}" -eq "true") {
174+
$credentialBytes = [Text.Encoding]::ASCII.GetBytes("admin:${{ inputs.admin-password }}")
175+
$encodedCredentials = [Convert]::ToBase64String($credentialBytes)
176+
$baseCredentials = "Basic $encodedCredentials"
177+
$Headers = @{ Authorization = $baseCredentials }
178+
$url = 'https://localhost:${{ inputs.port || 9200 }}/_cat/plugins'
179+
} else {
180+
$Headers = @{ }
181+
$url = 'http://localhost:${{ inputs.port || 9200 }}/_cat/plugins'
182+
}
183+
Invoke-WebRequest -SkipCertificateCheck -Uri $url -Headers $Headers;
184+
shell: pwsh
185+
working-directory: ${{ inputs.port }}
186+
187+
- if: always()
188+
run: cat ./opensearch-${{ inputs.opensearch-version }}-SNAPSHOT${{ inputs.port }}/logs/opensearch.log
189+
shell: bash
190+
working-directory: ${{ inputs.port }}
191+
192+
- if: always()
193+
run: cat ./opensearch-${{ inputs.opensearch-version }}-SNAPSHOT${{ inputs.port }}/config/opensearch.yml
194+
shell: bash
195+
working-directory: ${{ inputs.port }}

‎.github/workflows/release-e2e-workflow-template-windows.yml

+13
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,8 @@ jobs:
2525
# 2.12 onwards security demo configuration require a custom admin password
2626
OPENSEARCH_INITIAL_ADMIN_PASSWORD: 'myStrongPassword123!'
2727
steps:
28+
- name: Checkout Branch
29+
uses: actions/checkout@v3
2830
- name: Set up JDK
2931
uses: actions/setup-java@v1
3032
with:
@@ -69,6 +71,17 @@ jobs:
6971
fi
7072
netstat -anP tcp | grep LISTEN | grep 9200 || netstat -ntlp | grep 9200
7173
shell: bash
74+
- uses: ./.github/actions/start-opensearch
75+
with:
76+
opensearch-version: ${{ env.VERSION }}
77+
security-enabled: false
78+
port: 9201
79+
- uses: ./.github/actions/start-opensearch
80+
with:
81+
opensearch-version: ${{ env.VERSION }}
82+
security-enabled: true
83+
admin-password: admin
84+
port: 9202
7285
- name: Get and run OpenSearch-Dashboards
7386
run: |
7487
curl -SLO https://ci.opensearch.org/ci/dbc/distribution-build-opensearch-dashboards/${{ env.VERSION }}/latest/${{ env.PLATFORM }}/x64/zip/dist/opensearch-dashboards/opensearch-dashboards-${{ env.VERSION }}-${{ env.PLATFORM }}-x64.zip

‎.github/workflows/release-e2e-workflow-template.yml

+13
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,8 @@ jobs:
2828
# 2.12 onwards security demo configuration require a custom admin password
2929
OPENSEARCH_INITIAL_ADMIN_PASSWORD: 'myStrongPassword123!'
3030
steps:
31+
- name: Checkout Branch
32+
uses: actions/checkout@v3
3133
- name: Set up JDK
3234
uses: actions/setup-java@v1
3335
with:
@@ -66,6 +68,17 @@ jobs:
6668
timeout 900 bash -c 'while [[ "$(curl -o /dev/null -w ''%{http_code}'' -u admin:${{ env.OPENSEARCH_INITIAL_ADMIN_PASSWORD }} -k https://localhost:9200)" != "200" ]]; do sleep 5; done'
6769
curl https://localhost:9200 -u admin:${{ env.OPENSEARCH_INITIAL_ADMIN_PASSWORD }} --insecure
6870
fi
71+
- uses: ./.github/actions/start-opensearch
72+
with:
73+
opensearch-version: ${{ env.VERSION }}
74+
security-enabled: false
75+
port: 9201
76+
- uses: ./.github/actions/start-opensearch
77+
with:
78+
opensearch-version: ${{ env.VERSION }}
79+
security-enabled: true
80+
admin-password: admin
81+
port: 9202
6982
- name: Get OpenSearch-Dashboards
7083
run: |
7184
wget https://ci.opensearch.org/ci/dbc/distribution-build-opensearch-dashboards/${{ env.VERSION }}/latest/linux/x64/tar/dist/opensearch-dashboards/opensearch-dashboards-${{ env.VERSION }}-linux-x64.tar.gz

‎.github/workflows/release-signoff-chrome.yml

+11
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,17 @@ jobs:
3434
cd opensearch-${{ env.VERSION }}/
3535
./opensearch-tar-install.sh &
3636
timeout 900 bash -c 'while [[ "$(curl -s -o /dev/null -w ''%{http_code}'' -u admin:${{ env.OPENSEARCH_INITIAL_ADMIN_PASSWORD }} -k https://localhost:9200)" != "200" ]]; do sleep 5; done'
37+
- uses: ./.github/actions/start-opensearch
38+
with:
39+
opensearch-version: ${{ env.VERSION }}
40+
security-enabled: false
41+
port: 9201
42+
- uses: ./.github/actions/start-opensearch
43+
with:
44+
opensearch-version: ${{ env.VERSION }}
45+
security-enabled: true
46+
admin-password: admin
47+
port: 9202
3748
- name: Get OpenSearch-Dashboards
3849
run: |
3950
wget https://ci.opensearch.org/ci/dbc/distribution-build-opensearch-dashboards/${{ env.VERSION }}/latest/linux/x64/tar/dist/opensearch-dashboards/opensearch-dashboards-${{ env.VERSION }}-linux-x64.tar.gz

‎DEVELOPER_GUIDE.md

+38
Original file line numberDiff line numberDiff line change
@@ -126,6 +126,44 @@ Tests for plugins that are not a part of the [OpenSearch Dashboards](https://git
126126
/<YOUR_PLUGIN_NAME>
127127
```
128128

129+
### Tests for Multiple Datasources
130+
131+
Tests surrounding the multiple datasources feature can use the start-opensearch action that lives in this repo. Note that to test these related features, the following OSD config needs to be set: `osd-serve-args: --data_source.enabled=true`. Additionally, if testing with a remote datasource with basic auth enabled using this repo, an additional OSD config needs to be set: `osd-serve-args: --data_source.ssl.verificationMode: none`, so that the self-signed demo certificates can be used.
132+
133+
Example usage:
134+
```
135+
- uses: ./.github/actions/start-opensearch
136+
with:
137+
opensearch-version: 3.0.0
138+
security-enabled: false
139+
port: 9201
140+
```
141+
142+
143+
This will spin up an OpenSearch backend with version 3.0.0 on port 9201 within the same github runner. This OpenSearch can then be added as an datasource.
144+
145+
```
146+
- uses: ./.github/actions/start-opensearch
147+
with:
148+
opensearch-version: 3.0.0
149+
security-enabled: true
150+
admin-password: admin
151+
port: 9202
152+
```
153+
This will spin up an OpenSearch backend with version 3.0.0 on port 9202 with basic auth and admin credentials of "admin:admin" within the same github runner. This OpenSearch can then be added as an datasource.
154+
155+
To test UI/API compatibility with different versions you may want to spin up a matrix of OpenSearch backends with different versions than the local cluster. The earliest windows distribution supported is 2.4.
156+
157+
158+
The DataSourceManagement Plugin exposes a helper function to create a data source on this port:
159+
```
160+
const [noAuthId, noAuthLabel] = cy.createDataSourceNoAuth();
161+
162+
const [basicAuthId, basicAuthLabel] = cy.createDataSourceBasicAuth();
163+
164+
# Add tests that make calls using noAuthId and basicAuthId, or test that remote datasource via the UI using the labels noAuthLabel and basicAuthLabel.
165+
```
166+
129167
### Experimental Features
130168

131169
When writing tests for experimental features, please follow these steps.

‎cypress.json

+4
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,10 @@
1111
"viewportHeight": 1320,
1212
"env": {
1313
"openSearchUrl": "http://localhost:9200",
14+
"remoteDataSourceNoAuthUrl": "http://localhost:9201",
15+
"remoteDataSourceBasicAuthUrl": "https://localhost:9202",
16+
"remoteDataSourceBasicAuthUsername": "admin",
17+
"remoteDataSourceBasicAuthPassword": "admin",
1418
"SECURITY_ENABLED": false,
1519
"AGGREGATION_VIEW": false,
1620
"username": "admin",

‎cypress/integration/core-opensearch-dashboards/opensearch-dashboards/datasource-management-plugin/1_create_datasource.spec.js

+6
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,8 @@ const miscUtils = new MiscUtils(cy);
2323
const username = Cypress.env('username');
2424
const password = Cypress.env('password');
2525

26+
// TODO: create datasource with basic auth and sigv4
27+
2628
if (Cypress.env('DATASOURCE_MANAGEMENT_ENABLED')) {
2729
describe('Create datasources', () => {
2830
before(() => {
@@ -385,6 +387,10 @@ if (Cypress.env('DATASOURCE_MANAGEMENT_ENABLED')) {
385387
'app/management/opensearch-dashboards/dataSources'
386388
);
387389
});
390+
391+
it('creates a datasources to a real opensearch instance', () => {
392+
cy.createDataSourceNoAuth();
393+
});
388394
});
389395
});
390396
}

0 commit comments

Comments
 (0)