Add WarpX_CCACHE Option (#4637)

WarpX autodetects `ccache` and uses it, and there's nothing you can do
about it. This PR disables it by default, and lets developers enable it
through `-DWarpX_CCACHE:BOOL=ON`.

The reason for this is mostly related to compiler wrappers, of which
there are many...

If `g++` is a compiler wrapper, then `ccache g++` cannot see the
effective flags passed to underlying real `g++`. That leads eventually
to false positive cache hits, which is a pain to debug.

For compiler wrappers you want `g++` the wrapper to invoke the call to
`ccache <real g++ invocation>` to fix that, which is for example how
Spack handle it.

Further, if you use spack with ccache enabled, the defaults of WarpX
cause `ccache` to be invoked twice (inner & outer), doubling the cache
requirements.

Finally, compiler wrapper that handle ccache themselves may set further
ccache options / flags that WarpX does not set, such as disabling
hashing of the build dir -- w/o that option cache may be useless.

* Update Order and Docs

Author: haampie

CMakeLists.txt

@@ -43,7 +43,15 @@ set_cxx17_superbuild()
 # this is an optional tool that stores compiled object files; allows fast
 # re-builds even with "make clean" in between. Mainly used to store AMReX
 # objects
-set_ccache()
+if(CMAKE_SOURCE_DIR STREQUAL PROJECT_SOURCE_DIR)
+    set(WarpX_CCACHE_DEFAULT ON)
+else()
+    set(WarpX_CCACHE_DEFAULT OFF)  # we are a subproject in a superbuild
+endif()
+option(WarpX_CCACHE "Enable ccache for faster rebuilds" ${WarpX_CCACHE_DEFAULT})
+if(WarpX_CCACHE)
+    set_ccache()
+endif()
 
 
 # Output Directories ##########################################################
@@ -149,6 +157,7 @@ endif()
 # this defined the variable BUILD_TESTING which is ON by default
 #include(CTest)
 
+
 # Dependencies ################################################################
 #
 

Docs/source/install/cmake.rst

@@ -116,7 +116,7 @@ By default, the most important dependencies of WarpX are automatically downloade
 CMake Option                  Default & Values                               Description
 ============================= ============================================== ===========================================================
 ``BUILD_SHARED_LIBS``         ON/**OFF**                                     `Build shared libraries for dependencies <https://cmake.org/cmake/help/latest/variable/BUILD_SHARED_LIBS.html>`__
-``CCACHE_PROGRAM``            First found ``ccache`` executable.             Set to ``-DCCACHE_PROGRAM=NO`` to disable CCache.
+``WarpX_CCACHE``              **ON**/OFF                                     Search and use CCache to speed up rebuilds.
 ``AMReX_CUDA_PTX_VERBOSE``    ON/**OFF**                                     Print CUDA code generation statistics from ``ptxas``.
 ``WarpX_amrex_src``           *None*                                         Path to AMReX source directory (preferred if set)
 ``WarpX_amrex_repo``          ``https://github.com/AMReX-Codes/amrex.git``   Repository URI to pull and build AMReX from