Re-add 2x missing CMake IMPL_ flags to the Wiki

[felker]

Following a discussion with @masterleinad over Slack.

The BUILD.md in the Kokkos source repo used to list (all?) Kokkos_ prefixed CMake options, until April 2023 when it started to refer readers to the Wiki page: kokkos/kokkos@83873a6#diff-40f60e1037245d7b8a98a7325d53890a717da9979adeb54a61a795c4ba07f9c9R114

Both Kokkos_ENABLE_IMPL_HPX_ASYNC_DISPATCH and Kokkos_ENABLE_IMPL_CUDA_MALLOC_ASYNC weren't carried over to the Wiki at the time. The latter is especially confusing, given its changing default settings and issues with Cray MPICH on NERSC Perlmutter, ALCF Polaris, and LANL Venado, etc. kokkos/kokkos#7353

I have re-added them, plus added two other helpful CMake comments that @masterleinad had relayed to ALCF folks recently.

• Also update https://kokkos.org/kokkos-core-wiki/known-issues.html#cuda and add a mention of the Cray MPICH issues with async malloc on CUDA?

I manually diff'd the pre-April 2023 BUILD.md and the current Wiki page, and found several other flags that also didnt make the transition. Not sure if all of them were removed or deprecated, but here they are:

* Kokkos_ENABLE_IMPL_HPX_ASYNC_DISPATCH
    * Whether HPX supports asynchronous dispatch
    * BOOL Default: ON
* Kokkos_ENABLE_IMPL_CUDA_MALLOC_ASYNC
    * Whether to enable CudaMallocAsync (requires CUDA Toolkit 11.2). This is an experimental performance feature and currently has issue when using with UCX. See https://github.com/kokkos/kokkos/issues/4228 for more details.
    * BOOL Default: OFF

* Kokkos_ENABLE_PROFILING_LOAD_PRINT
    * Whether to print information about which profiling tools gotloaded
    * BOOL Default: OFF

* Kokkos_ENABLE_LIBNUMA
    * Whether to enable the LIBNUMA library
    * BOOL Default: Off
* Kokkos_ENABLE_MEMKIND
    * Whether to enable the MEMKIND library
    * BOOL Default: Off
* Kokkos_ENABLE_LIBRT
    * Whether to enable the LIBRT library
    * BOOL Default: Off

* Kokkos_LIBNUMA_DIR or LIBNUMA_ROOT
    * Location of LIBNUMA install prefix
    * PATH Default:
* Kokkos_MEMKIND_DIR or MEMKIND_ROOT
    * Location of MEMKIND install prefix
    * PATH Default:
* Kokkos_LIBRT_DIR or LIBRT_ROOT
    * Location of LIBRT install prefix
    * PATH Default:

[dalg24]

Their removal was not an accident but I agree that we need to reintroduce IMPL_CUDA_MALLOC_ASYNC.

I am fine with mentioning IMPL_HPX_ASYNC_DISPATCH as well.

[dalg24]

can fundamentally alter the underlying implementation of Kokkos on a given backend

We should make it a general warning right after the note suggesting that users can use ccmake to discover the list of options. It is not specific to backend options. Essentially, anything that says IMPL is considered private implementation detail and we reserve the right to change at anytime without prior notice.

@nliber would you please add a section on CMake variables in the compatibility guidelines

[felker]

Fixed all issues