Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

GlutenConfig should using SQLConf provided by SparkSession first #7690

Open
beliefer opened this issue Oct 26, 2024 · 0 comments · May be fixed by #8962
Open

GlutenConfig should using SQLConf provided by SparkSession first #7690

beliefer opened this issue Oct 26, 2024 · 0 comments · May be fixed by #8962
Labels
enhancement New feature or request

Comments

@beliefer
Copy link
Contributor

beliefer commented Oct 26, 2024 •
edited
Loading

Description

GlutenConfig wrapped the immutable SQLConf instance, this lead to the dynamically modified configurations in interactive scenarios can't apply to runtime.
Take for an example.

scala > SET spark.gluten.sql.enable.native.validation = false;
...
scala > SELECT ...
...
SET spark.gluten.sql.enable.native.validation = true;
...
scala > SELECT ...
...

We should using SQLConf provided by SparkSession first
@beliefer beliefer added the enhancement New feature or request label Oct 26, 2024
@beliefer beliefer changed the title Remove GlutenConfig from ColumnarRuleApplier GlutenConfig should supports config change at runtime Oct 28, 2024
@beliefer beliefer changed the title GlutenConfig should supports config change at runtime GlutenConfig should support runtime configuration changes Oct 28, 2024
@github-actions github-actions bot mentioned this issue Oct 28, 2024
Closed
@beliefer beliefer mentioned this issue Nov 6, 2024
Closed
@beliefer beliefer closed this as completed Nov 7, 2024
@beliefer beliefer changed the title GlutenConfig should support runtime configuration changes GlutenConfig should using SQLConf provided by SparkSession first Mar 11, 2025
@beliefer beliefer reopened this Mar 11, 2025
@beliefer beliefer linked a pull request Mar 11, 2025 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
1 participant
@beliefer