[VL] Allow to reuse local SSD cache on Spark context restart

[zhouyuan]

Description

Currently local SSD cache will be discarded after Spark context shutdown. It would better to make it reusable with a new optional config

[jackylee-ch]

Curious about the scenarios where reuse would be necessary. In a cluster environment, if a Spark Application is regenerated, the executors will be re-allocated. The cache on the executors may not align, and even if there is cache, the hit rate might not be high.

[zhouyuan]

@jackylee-ch Thanks, I was told the soft cache affinity can help to alleviate this issue, but I'm trying to find more resource verify
https://github.com/apache/incubator-gluten/blob/main/shims/common/src/main/scala/org/apache/gluten/config/GlutenConfig.scala#L691

[jackylee-ch]

I was told the soft cache affinity can help to alleviate this issue, but I'm trying to find more resource verify

AFAIK, the cache involved in duplicateReading will become invalid after Spark restarts, unless the cache can be reused and the executor rescheduling issue is resolved. Nevertheless, the pr is still helpful for cache reuse within the same application.

[FelixYBW]

@jackylee-ch It's more for benchmark testing actually. So the warm run in second test can get 100% hit.
You are right in production the local ssd cache may be used in the same query but hard to in second query.

[zhouyuan]

For single instance based tests it should be still useful.