Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remove sparkr and any references to it #160

Open
StrongestNumber9 opened this issue Jan 28, 2025 · 0 comments · May be fixed by #179
Open

Remove sparkr and any references to it #160

StrongestNumber9 opened this issue Jan 28, 2025 · 0 comments · May be fixed by #179
Labels
enhancement New feature or request

Comments

@StrongestNumber9
Copy link
Contributor

Description

Remove sparkr loading from

Remove sparkr and any references to it

Use case or motivation behind the feature request

We do not use or support R. While we do not build or ship sparkr in this project, there is no reason upload it in every single spark application launch as it is just extra weight for no reason.

Including, but limited to, this part.

zep_01/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/launcher/SparkInterpreterLauncher.java

Lines 344 to 369 in e233bda

private void setupPropertiesForSparkR(Properties sparkProperties) {
if (isYarnMode()) {
String sparkHome = getEnv("SPARK_HOME");
File sparkRBasePath = null;
if (sparkHome == null) {
if (!getSparkMaster().startsWith("local")) {
throw new RuntimeException("SPARK_HOME is not specified in interpreter-setting" +
" for non-local mode, if you specify it in zeppelin-env.sh, please move that into " +
" interpreter setting");
}
String zeppelinHome = zConf.getString(ZeppelinConfiguration.ConfVars.ZEPPELIN_HOME);
sparkRBasePath = new File(zeppelinHome,
"interpreter" + File.separator + "spark" + File.separator + "R");
} else {
sparkRBasePath = new File(sparkHome, "R" + File.separator + "lib");
}
File sparkRPath = new File(sparkRBasePath, "sparkr.zip");
if (sparkRPath.exists() && sparkRPath.isFile()) {
mergeSparkProperty(sparkProperties, "spark.yarn.dist.archives",
sparkRPath.getAbsolutePath() + "#sparkr");
} else {
LOGGER.warn("sparkr.zip is not found, SparkR may not work.");
}
}
}

Related issues

Additional context
@StrongestNumber9 StrongestNumber9 added the enhancement New feature or request label Jan 28, 2025
@StrongestNumber9 StrongestNumber9 linked a pull request Feb 10, 2025 that will close this issue
Remove sparkr and related loading code, partially re-open SparkInterp… #179
Open
33 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Remove sparkr and related loading code, partially re-open SparkInterp… StrongestNumber9/zep_01
1 participant
@StrongestNumber9