[Bug] [Yarn-perjob] NoSuchMethodError: org.apache.commons.cli.Option.builder(Ljava/lang/String;)Lorg/apache/commons/cli/Option$Builder;

[RainBow-Suger]

Search before asking

• I had searched in the issues and found no similar issues.

What happened

I encountered an error while using Yarn's per job and application execution mode
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.commons.cli.Option.builder(Ljava/lang/String;)Lorg/apache/commons/cli/Option$Builder;
at org.apache.flink.runtime.entrypoint.parser.CommandLineOptions.(CommandLineOptions.java:27)
at org.apache.flink.runtime.entrypoint.DynamicParametersConfigurationParserFactory.options(DynamicParametersConfigurationParserFactory.java:43)
at org.apache.flink.runtime.entrypoint.DynamicParametersConfigurationParserFactory.getOptions(DynamicParametersConfigurationParserFactory.java:50)
at org.apache.flink.runtime.entrypoint.parser.CommandLineParser.parse(CommandLineParser.java:42)
at org.apache.flink.runtime.entrypoint.ClusterEntrypointUtils.parseParametersOrExit(ClusterEntrypointUtils.java:70)
at org.apache.flink.yarn.entrypoint.YarnJobClusterEntrypoint.main(YarnJobClusterEntrypoint.java:99)
The official website explains that the per job and application execution mode of Dinky's current version of yarn depend on flink-shaded-Hadoop, and an additional flink-shaded-Hadoop-uber package needs to be added. If you are using flink-shaded-Hadoop-uber-3, please manually remove the conflicting content such as Javax.Servlet inside the package. I did add the flink-shaded-hadoop user package, and the version is 3.1.1.7.2.90-173-9.0. However, the error NoSuchMethodError still occurs: org.apache.commons.cli.Option.builder(Ljava/lang/String;)Lorg/apache/commons/cli/Option$Builder
I executed 'for i in $(ls *. jar); do echo $i && jar tf $i | grep org.apache.commons.cli.Options.class; done' in the dinky/extends directory and found that the flink-dist and flink-shaded-hadoop-3-uber packages used org.apache.commons.cli.Option.class
This class exists under the dependencies of commons-cli, and flink-dist uses the 1.5.0 version of commons-cli; flink-shaded-hadoop-3-uber uses version 1.2 of commons-cli, which is a lower version

What you expected to happen

I hope to upgrade the commons-cli in flink-shaded-hadoop-uber-3 to version 1.5.0

How to reproduce

After adding Uber dependency, Dinky executes according to Yarn's perjob mode dinky version: 1.2.1 flink: 1.20 hadoop: 3.3.4

Anything else

No response

Version

1.2.1

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[github-actions]

Hello @RainBow-Suger, this issue is about web, so I assign it to @Zzm0809. If you have any questions, you can comment and reply.

你好 @RainBow-Suger, 这个 issue 是关于 web 的，所以我把它分配给了 @Zzm0809。如有任何问题，可以评论回复。

[Zzm0809]

flink-shaded-hadoop-uber-3 的 common-cli dinky 升级不了 可以在 extends 下放一个 common-cli 的合适的版本的 jar

[RainBow-Suger]

按照你的方法，在我家里的环境中测试不行，还是报相同错误；我采用dinky-client-hadoop pom，剔除了其中commons-cli 1.2版本并重新引入commons-cli 1.5.0版本，再重新编译，扔到公司的测试服务器上是可行的，且我把公司dinky下载下来，放到我家里的测试服务器上依然报相同错，我不是很理解