[CORE][umbrella] Configuration enhancements

[yikf]

Description

As discussed in #6970, we plan to enhance the gluten configuration, and the specific enhancement solutions can be found in section #6970 (comment).

This issue serves as an umbrella to record the task list that we need to include to achieve our goals, because enhance config involves a large number of files, splitting multiple tasks facilitates code review and reduces the risk of introducing other problems.

1. [CORE] Introduce the ConfigEntry to make the config definition more flexible #8327 This task involves a large number of files. Due to the performance regressions of the CH backend, we refined this task into three parts:

• [GLUTEN-8327][CORE][Part-1] Rename GlutenConfig.getConf to GlutenConfig.get #8395
• [GLUTEN-8327][CORE][Part-2] Move GlutenConfig.scala to org.apache.gluten.config package dir  #8426
• [GLUTEN-8327][CORE][Part-3] Introduce the ConfigEntry to gluten config #8431

2. [CORE] Split the configuration into individual modules that belong to each other. #8479

• [GLUTEN-8479][CORE][Part-1] Remove unnecessary config #8480
• [GLUTEN-8479][CORE][Part-2] All configurations should be defined through ConfigEntry #8559
• [GLUTEN-8479][CORE][Part-3] Split backend configs to its corresponding modules #8586

3. Split static config into independent file.

• [GLUTEN-8919] Split static config into independent file #8920

5. Standardize configuration naming, log the deprecated config name if used.
6. [CORE] Support generate configurations golden file and check it  #8070

[baibaichen]

@yikf as discussed in the offline, there are 2 requirement

1. Pass spark config to backend. For example, ch backend need spark.sql.caseSensitive to choose compare behavior.
2. Pass value type to backend. currently we use ConfigMap which assum value is string type.

[FelixYBW]

@yikf Is the enhancement finished? Is there a readme about how to add a configuration?

[yikf]

@yikf Is the enhancement finished? Is there a readme about how to add a configuration?

It's not completed yet. I will keep following up in the next few days.

Eventually, there will be a configuration.md file to describe the configuration. Regarding the description of how to add the configuration, I can add it in this file.