[BUG] OpenSearch 2.19.0 with installed repository-s3 plugin cannot be stopped

[olfuerniss]

Describe the bug

When the repository-s3 plugin is installed, it is no longer possible to stop a running OpenSearch instance. Happens since 2.19.0.

Related component

Plugins

To Reproduce

1. Start OpenSearch with installed 'repository-s3' plugin

curl -O https://artifacts.opensearch.org/releases/bundle/opensearch/2.19.0/opensearch-2.19.0-linux-x64.tar.gz
tar xvfz opensearch-2.19.0-linux-x64.tar.gz
cd opensearch-2.19.0/
bin/opensearch-plugin install repository-s3
echo 'plugins.security.disabled: true' >> config/opensearch.yml
bin/opensearch

2. now wait until OpenSearch has started ... and try to stop it by pressing CONTROL/STRG+c

=> It does not stop, shows the following exception and hangs (must be killed)

[2025-02-14T11:02:16,876][ERROR][i.n.u.c.D.rejectedExecution] [Mac.lan] Failed to submit a listener notification task. Event loop shut down?
java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "getClassLoader")
	at java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:485) ~[?:?]
	at java.base/java.security.AccessController.checkPermission(AccessController.java:1068) ~[?:?]
	at java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:416) ~[?:?]
	at java.base/java.lang.ClassLoader.checkClassLoaderPermission(ClassLoader.java:2060) ~[?:?]
	at java.base/java.lang.Thread.getContextClassLoader(Thread.java:1502) ~[?:?]
	at io.netty.util.concurrent.GlobalEventExecutor.startThread(GlobalEventExecutor.java:239) ~[netty-common-4.1.117.Final.jar:4.1.117.Final]
	at io.netty.util.concurrent.GlobalEventExecutor.execute0(GlobalEventExecutor.java:232) ~[netty-common-4.1.117.Final.jar:4.1.117.Final]
	at io.netty.util.concurrent.GlobalEventExecutor.execute(GlobalEventExecutor.java:226) ~[netty-common-4.1.117.Final.jar:4.1.117.Final]
	at io.netty.util.concurrent.DefaultPromise.safeExecute(DefaultPromise.java:862) [netty-common-4.1.117.Final.jar:4.1.117.Final]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:500) [netty-common-4.1.117.Final.jar:4.1.117.Final]
	at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) [netty-common-4.1.117.Final.jar:4.1.117.Final]
	at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) [netty-common-4.1.117.Final.jar:4.1.117.Final]
	at io.netty.util.concurrent.DefaultPromise.setSuccess(DefaultPromise.java:97) [netty-common-4.1.117.Final.jar:4.1.117.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:1059) [netty-common-4.1.117.Final.jar:4.1.117.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.117.Final.jar:4.1.117.Final]
	at java.base/java.lang.Thread.run(Thread.java:840) [?:?]

Expected behavior

OpenSearch should not hang on stop when the repository-s3 plugin is installed.

Additional Details

Plugins

• all plugins of 2.19.0
• repository-s3

Host/Environment (please complete the following information):

• OS: macOS Sequoia
• Version 15.3.1

[spapadop]

faced exactly the same just after the upgrade (running on AlmaLinux9 and installing using the RPM)

[HenryTheSir]

Can confirm on RHEL8 with tarball installation and added s3 plugin

[cwperks]

Looks like the same stacktrace from netty/netty#14743

@reta will the netty upgrade to 4.1.118.Final resolve this issue?

[reta]

Thanks @cwperks

Looks like the same stacktrace from netty/netty#14743

Yes

@reta will the https://github.com/opensearch-project/OpenSearch/pull/17320resolve this issue?

Yes

@spapadop @HenryTheSir @olfuerniss could you folks please confirm if the issue is not happening when repository-s3 plugin is not installed?

[olfuerniss]

@reta Yes

[spapadop]

yeap I double-confirm, everything seems happy without repository-s3.

[reta]

The 2.19.0 didn't get Netty 4.1.118 (sadly), listing possible options here:

• 2.19.1 patch release (best option)
• build and publish s3-repository plugin locally
• in place patching (replace Netty 4.1.117 with 4.1.118 JARs)

@andrross any other options I am missing? thank you

[andrross]

Yeah, it looks like a 2.19.1 patch release is probably the way to go here. FYI @getsaurabh02

It's also worth thinking about how we missed this in testing.

[getsaurabh02]

Thank you @reta and @andrross for sharing the details. I also agree that patching 2.19 with fix and releasing 2.19.1 is the best option currently. Option 2 of building the repository-s3 plugin locally and publishing might be tricky and we will like to skip.

Now, since the issue occurs while stoping a running OpenSearch instance, I will like to discuss the urgency of the patch release. Based on the current release schedule, the 2.19.1 patch is planned for April 08, do we need expedite this patch release or should we wait as per the current schedule?

Also, what are the functional and usability issues which OS users might run into due to this bug. Adding more folks into the discussion to arrive on the patch release plan - @cwperks @msfroh @Pallavi-AWS @dblock

On the miss for tests, @andrross @reta should we open an issue and follow up to see how we can add more Integ and Smoke Test for build and distrubtion

[getsaurabh02]

Adding @petardz @gaiksaya @Divyaasm from Infra for visibility.

[rishabh6788]

Adding @petardz @gaiksaya @Divyaasm from Infra for visibility.

tagging correct @peterzhuamazon :)

[peterzhuamazon]

Since distribution pull the plugin from s3 it is easier to make a quick fix there.
But that means manual steps instead of automation, and leave maven untouched with the old version.

Thanks.

[reta]

Since distribution pull the plugin from s3 it is easier to make a quick fix there. But that means manual steps instead of automation, and leave maven untouched with the old version.

I think we could not update the published binaries without the version bump, right?

On the miss for tests, @andrross @reta should we open an issue and follow up to see how we can add more Integ and Smoke Test for build and distrubtion

We definitely need to figure out how to test that.

[gaiksaya]

I think we could not update the published binaries without the version bump, right?

Yes! Maven does not allow overwriting the same version binaries. Recommend to do a patch instead to avoid having different artifacts on different platforms.

[cwperks]

I'm not sure if its possible with the current tests in the repository-s3 module, but ISM has interesting test cleanup logic that checks for running threads and I have seen it catch issues like this in the past.

[reta]

I'm not sure if its possible with the current tests in the repository-s3 module, but ISM has interesting test cleanup logic that checks for running threads and I have seen it catch issues like this in the past.

AFAIK, all OpenSearch based tests check for thread leaks, but for this issue, none of the tests have caught it sadly (affected plugins included transport-netty4, transport-reactor-netty4 and s3-repository)

[moesterheld]

Facing the same problem. Since we use scripts to start/stop OS, this is a blocker for us to update.
I will try to replace netty locally to see if that helps me as a workaround

[Divyaasm]

2.19.1 Release Issue can be tracked here

[shpatserman]

Same problem

[]# dnf list installed opensearch
Updating Subscription Management repositories.
Installed Packages
opensearch.x86_64                                                                                                                   2.19.0-1                                                                                                                   @opensearch-2.x
[]# uname -a
Linux [] 5.14.0-362.24.2.el9_3.x86_64 #1 SMP PREEMPT_DYNAMIC Sat Mar 30 14:11:54 EDT 2024 x86_64 x86_64 x86_64 GNU/Linux
[]#

[peterzhuamazon]

Adding @krisfreedain @nateynateynate @kolchfa-aws that 2.19.1 is coming next week.

[Divyaasm]

Tested the first RC of 2.19.1 to confirm if the issue is resolved. LGTM! . Find the RC details here for testing.

`[2025-02-24T17:37:34,361][INFO` ][o.o.i.i.m.ISMTemplateService] [bcd07416bf93] Successfully update template migration setting
^C[2025-02-24T17:37:51,761][INFO ][o.o.s.a.r.AuditMessageRouter] [bcd07416bf93] Closing AuditMessageRouter
[2025-02-24T17:37:51,761][INFO ][o.o.n.Node               ] [bcd07416bf93] stopping ...
[2025-02-24T17:37:51,763][INFO ][o.o.s.a.s.SinkProvider   ] [bcd07416bf93] Closing InternalOpenSearchSink
[2025-02-24T17:37:51,763][INFO ][o.o.s.a.s.SinkProvider   ] [bcd07416bf93] Closing DebugSink
[2025-02-24T17:37:51,910][INFO ][o.o.n.Node               ] [bcd07416bf93] stopped
[2025-02-24T17:37:51,910][INFO ][o.o.n.Node               ] [bcd07416bf93] closing ...
[2025-02-24T17:37:51,913][INFO ][o.o.s.a.i.AuditLogImpl   ] [bcd07416bf93] Closing AuditLogImpl
[2025-02-24T17:37:51,919][INFO ][o.o.n.Node               ] [bcd07416bf93] closed

[peterzhuamazon]

Hi @olfuerniss @spapadop @HenryTheSir @moesterheld @shpatserman @reta could you please help confirm the above Release Candidates posted by @Divyaasm solves the issues?

We will go ahead with a official release on 2025/02/27.

Thanks!

[reta]

Hi @olfuerniss @spapadop @HenryTheSir @moesterheld @shpatserman @reta could you please help confirm the above Release Candidates posted by @Divyaasm solves the issues?

We will go ahead with a official release on 2025/02/27.

Thanks!

Yes, the issue is resolved, thanks @peterzhuamazon

[olfuerniss]

@peterzhuamazon Sorry, can't test it as it is. I do get an 403 (access denied) when I try to install the repository-s3 plugin (OpenSearch 2.19.1):

bin/opensearch-plugin install repository-s3 -> Installing repository-s3 -> Downloading repository-s3 from opensearch -> Failed installing repository-s3 -> Rolling back repository-s3 -> Rolled back repository-s3 Exception in thread "main" java.io.IOException: Server returned HTTP response code: 403 for URL: https://artifacts.opensearch.org/releases/plugins/repository-s3/2.19.1/repository-s3-2.19.1.zip at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:2037) at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1626) at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:224) at org.opensearch.plugins.InstallPluginCommand.downloadZip(InstallPluginCommand.java:434) at org.opensearch.plugins.InstallPluginCommand.downloadAndValidate(InstallPluginCommand.java:511) at org.opensearch.plugins.InstallPluginCommand.download(InstallPluginCommand.java:310) at org.opensearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:271) at org.opensearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:248) at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) at org.opensearch.cli.MultiCommand.execute(MultiCommand.java:104) at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) at org.opensearch.cli.Command.main(Command.java:101) at org.opensearch.plugins.PluginCli.main(PluginCli.java:60)

[peterzhuamazon]

@peterzhuamazon Sorry, can't test it as it is. I do get an 403 (access denied) when I try to install the repository-s3 plugin (OpenSearch 2.19.1):

bin/opensearch-plugin install repository-s3 -> Installing repository-s3 -> Downloading repository-s3 from opensearch -> Failed installing repository-s3 -> Rolling back repository-s3 -> Rolled back repository-s3 Exception in thread "main" java.io.IOException: Server returned HTTP response code: 403 for URL: https://artifacts.opensearch.org/releases/plugins/repository-s3/2.19.1/repository-s3-2.19.1.zip at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:2037) at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1626) at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:224) at org.opensearch.plugins.InstallPluginCommand.downloadZip(InstallPluginCommand.java:434) at org.opensearch.plugins.InstallPluginCommand.downloadAndValidate(InstallPluginCommand.java:511) at org.opensearch.plugins.InstallPluginCommand.download(InstallPluginCommand.java:310) at org.opensearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:271) at org.opensearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:248) at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) at org.opensearch.cli.MultiCommand.execute(MultiCommand.java:104) at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) at org.opensearch.cli.Command.main(Command.java:101) at org.opensearch.plugins.PluginCli.main(PluginCli.java:60)

That is expected since it is a RC version not official release: opensearch-project/opensearch-build#5152 (comment)

Could you try to manually download the zip and install as well?
https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.19.1/10855/linux/x64/tar/builds/opensearch/core-plugins/repository-s3-2.19.1.zip

Thanks.

[olfuerniss]

LGTM :-)

curl -O https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.19.1/10855/linux/x64/tar/builds/opensearch/core-plugins/repository-s3-2.19.1.zip
curl -O https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.19.1/10855/linux/x64/tar/dist/opensearch/opensearch-2.19.1-linux-x64.tar.gz
tar xvfz opensearch-2.19.1-linux-x64.tar.gz
cd opensearch-2.19.1/
bin/opensearch-plugin install file:///Users/of/tmp/repository-s3-2.19.1.zip
echo 'plugins.security.disabled: true' >> config/opensearch.yml
bin/opensearch

Check installed plugins

curl 'localhost:9200/_cat/plugins'

Mac.lan opensearch-alerting                  2.19.1.0
Mac.lan opensearch-anomaly-detection         2.19.1.0
Mac.lan opensearch-asynchronous-search       2.19.1.0
Mac.lan opensearch-cross-cluster-replication 2.19.1.0
Mac.lan opensearch-custom-codecs             2.19.1.0
Mac.lan opensearch-flow-framework            2.19.1.0
Mac.lan opensearch-geospatial                2.19.1.0
Mac.lan opensearch-index-management          2.19.1.0
Mac.lan opensearch-job-scheduler             2.19.1.0
Mac.lan opensearch-knn                       2.19.1.0
Mac.lan opensearch-ltr                       2.19.1.0
Mac.lan opensearch-ml                        2.19.1.0
Mac.lan opensearch-neural-search             2.19.1.0
Mac.lan opensearch-notifications             2.19.1.0
Mac.lan opensearch-notifications-core        2.19.1.0
Mac.lan opensearch-observability             2.19.1.0
Mac.lan opensearch-performance-analyzer      2.19.1.0
Mac.lan opensearch-reports-scheduler         2.19.1.0
Mac.lan opensearch-security                  2.19.1.0
Mac.lan opensearch-security-analytics        2.19.1.0
Mac.lan opensearch-skills                    2.19.1.0
Mac.lan opensearch-sql                       2.19.1.0
Mac.lan opensearch-system-templates          2.19.1.0
Mac.lan query-insights                       2.19.1.0
Mac.lan repository-s3                        2.19.1

AND ... now it can be stopped with Control+C. Thanks!!!

[peterzhuamazon]

Thanks @olfuerniss then we are good to release on Thursday!

[reta]

Closing this one, thanks folks!

[AndreVirtimo]

Why is this issue not mentioned in the release notes?

[reta]

Why is this issue not mentioned in the release notes?

The issue is caused by Netty and has zero changes on OpenSearch side besides dependency update - which is in release notes.

[AndreVirtimo]

I understand this point of view. But the effect on the user is different. The users are affected by this bug and they have no reason why this dependency update should help them. The release notes are not (in my opinion) a change log for other developers, they are an important information for the users. When I ran into that bug the release notes should give me a hint that this update would help. But currently they need to try or to find this GH issue.