FEAT-modin-project#6735: Make Modin on MPI through unidist component more obvious (modin-project#6736)

Signed-off-by: Igoshev, Iaroslav <iaroslav.igoshev@intel.com>

Author: YarShev

.github/workflows/ci-notebooks.yml

@@ -8,6 +8,7 @@ on:
       - setup.cfg
       - setup.py
       - requirements/env_hdk.yml
+      - requirements/env_unidist_linux.yml
 concurrency:
   # Cancel other jobs in the same branch. We don't care whether CI passes
   # on old commits.
@@ -28,12 +29,17 @@ jobs:
     steps:
       - uses: actions/checkout@v3
       - uses: ./.github/actions/python-only
-        if: matrix.execution != 'hdk_on_native'
+        if: matrix.execution != 'hdk_on_native' && matrix.execution != 'pandas_on_unidist'
       - uses: ./.github/actions/mamba-env
         with:
           environment-file: requirements/env_hdk.yml
           activate-environment: modin_on_hdk
         if: matrix.execution == 'hdk_on_native'
+      - uses: ./.github/actions/mamba-env
+        with:
+          environment-file: requirements/env_unidist_linux.yml
+          activate-environment: modin_on_unidist
+        if: matrix.execution == 'pandas_on_unidist'
       - name: Cache datasets
         uses: actions/cache@v2
         with:
@@ -43,29 +49,29 @@ jobs:
       # replace modin with . in the tutorial requirements file for `pandas_on_ray` and
       # `pandas_on_dask` since we need Modin built from sources
       - run: sed -i 's/modin/./g' examples/tutorial/jupyter/execution/${{ matrix.execution }}/requirements.txt
-        if: matrix.execution != 'hdk_on_native'
+        if: matrix.execution != 'hdk_on_native' && matrix.execution != 'pandas_on_unidist'
       # install dependencies required for notebooks execution for `pandas_on_ray` and `pandas_on_dask`
       # Override modin-spreadsheet install for now
       - run: |
           pip install -r examples/tutorial/jupyter/execution/${{ matrix.execution }}/requirements.txt
           pip install git+https://github.com/modin-project/modin-spreadsheet.git@49ffd89f683f54c311867d602c55443fb11bf2a5
-        if: matrix.execution != 'hdk_on_native'
-      # Build Modin from sources for `hdk_on_native`
+        if: matrix.execution != 'hdk_on_native' && matrix.execution != 'pandas_on_unidist'
+      # Build Modin from sources for `hdk_on_native` and `pandas_on_unidist`
       - run: pip install -e .
-        if: matrix.execution == 'hdk_on_native'
+        if: matrix.execution == 'hdk_on_native' || matrix.execution == 'pandas_on_unidist'
       # install test dependencies
       # NOTE: If you are changing the set of packages installed here, make sure that
       # the dev requirements match them.
       - run: pip install pytest pytest-cov black flake8 flake8-print flake8-no-implicit-concat
-        if: matrix.execution != 'hdk_on_native'
+        if: matrix.execution != 'hdk_on_native' && matrix.execution != 'pandas_on_unidist'
       - run: pip install flake8-print jupyter nbformat nbconvert
-        if: matrix.execution == 'hdk_on_native'
+        if: matrix.execution == 'hdk_on_native' || matrix.execution == 'pandas_on_unidist'
       - run: pip list
-        if: matrix.execution != 'hdk_on_native'
+        if: matrix.execution != 'hdk_on_native' && matrix.execution != 'pandas_on_unidist'
       - run: |
           conda info
           conda list
-        if: matrix.execution == 'hdk_on_native'
+        if: matrix.execution == 'hdk_on_native' || matrix.execution == 'pandas_on_unidist'
       # setup kernel configuration for `pandas_on_unidist` execution with mpi backend
       - run: python examples/tutorial/jupyter/execution/${{ matrix.execution }}/setup_kernel.py
         if: matrix.execution == 'pandas_on_unidist'

.github/workflows/ci.yml

@@ -93,11 +93,17 @@ jobs:
       - uses: actions/checkout@v3
       - uses: ./.github/actions/python-only
       - run: python -m pip install -e ".[all]"
-      - name: Ensure all engines start up
+      - name: Ensure Ray and Dask engines start up
         run: |
           MODIN_ENGINE=dask python -c "import modin.pandas as pd; print(pd.DataFrame([1,2,3]))"
           MODIN_ENGINE=ray python -c "import modin.pandas as pd; print(pd.DataFrame([1,2,3]))"
+      - name: Ensure MPI engine start up
+        # Install a working MPI implementation beforehand so mpi4py can link to it
+        run: |
+          sudo apt install libmpich-dev
+          python -m pip install -e ".[mpi]"
           MODIN_ENGINE=unidist UNIDIST_BACKEND=mpi mpiexec -n 1 python -c "import modin.pandas as pd; print(pd.DataFrame([1,2,3]))"
+        if: matrix.os == 'ubuntu'
 
   test-internals:
     needs: [lint-flake8, lint-black-isort]

README.md

@@ -57,24 +57,30 @@ The charts below show the speedup you get by replacing pandas with Modin based o
 Modin can be installed with `pip` on Linux, Windows and MacOS:
 
 ```bash
-pip install "modin[all]" # (Recommended) Install Modin with all of Modin's currently supported engines.
+pip install "modin[all]" # (Recommended) Install Modin with Ray and Dask engines.
 ```
 
 If you want to install Modin with a specific engine, we recommend:
 
 ```bash
 pip install "modin[ray]" # Install Modin dependencies and Ray.
 pip install "modin[dask]" # Install Modin dependencies and Dask.
-pip install "modin[unidist]" # Install Modin dependencies and Unidist.
+pip install "modin[mpi]" # Install Modin dependencies and MPI through unidist.
 ```
 
+To get Modin on MPI through unidist (as of unidist 0.5.0) fully working
+it is required to have a working MPI implementation installed beforehand.
+Otherwise, installation of `modin[mpi]` may fail. Refer to
+[Installing with pip](https://unidist.readthedocs.io/en/latest/installation.html#installing-with-pip)
+section of the unidist documentation for more details about installation.
+
 Modin automatically detects which engine(s) you have installed and uses that for scheduling computation.
 
 #### From conda-forge
 
 Installing from [conda forge](https://github.com/conda-forge/modin-feedstock) using `modin-all`
 will install Modin and four engines: [Ray](https://github.com/ray-project/ray), [Dask](https://github.com/dask/dask),
-[Unidist](https://github.com/modin-project/unidist) and [HDK](https://github.com/intel-ai/hdk).
+[MPI through unidist](https://github.com/modin-project/unidist) and [HDK](https://github.com/intel-ai/hdk).
 
 ```bash
 conda install -c conda-forge modin-all
@@ -85,10 +91,14 @@ Each engine can also be installed individually (and also as a combination of sev
 ```bash
 conda install -c conda-forge modin-ray  # Install Modin dependencies and Ray.
 conda install -c conda-forge modin-dask # Install Modin dependencies and Dask.
-conda install -c conda-forge modin-unidist # Install Modin dependencies and Unidist.
+conda install -c conda-forge modin-mpi # Install Modin dependencies and MPI through unidist.
 conda install -c conda-forge modin-hdk # Install Modin dependencies and HDK.
 ```
 
+Refer to
+[Installing with conda](https://unidist.readthedocs.io/en/latest/installation.html#installing-with-conda)
+section of the unidist documentation for more details on how to install a specific MPI implementation to run on.
+
 To speed up conda installation we recommend using libmamba solver. To do this install it in a base environment:
 
 ```bash
@@ -119,7 +129,7 @@ export MODIN_ENGINE=unidist # Modin will use Unidist
 ```
 
 If you want to choose the Unidist engine, you should set the additional environment 
-variable ``UNIDIST_BACKEND``, because currently Modin only supports Unidist on MPI:
+variable ``UNIDIST_BACKEND``. Currently, Modin only supports MPI through unidist:
 
 ```bash
 export UNIDIST_BACKEND=mpi # Unidist will use MPI backend
@@ -144,7 +154,7 @@ _Note: You should not change the engine after your first operation with Modin as
 
 #### Which engine should I use?
 
-On Linux, MacOS, and Windows you can install and use either Ray, Dask or Unidist. There is no knowledge required
+On Linux, MacOS, and Windows you can install and use either Ray, Dask or MPI through unidist. There is no knowledge required
 to use either of these engines as Modin abstracts away all of the complexity, so feel
 free to pick either!
 

docs/conf.py

@@ -26,7 +26,16 @@ def noop_decorator(*args, **kwargs):
 ray.remote = noop_decorator
 
 # fake modules if they're missing
-for mod_name in ("cudf", "cupy", "pyarrow.gandiva", "pyhdk", "pyhdk.hdk", "xgboost"):
+for mod_name in (
+    "cudf",
+    "cupy",
+    "pyarrow.gandiva",
+    "pyhdk",
+    "pyhdk.hdk",
+    "xgboost",
+    "unidist",
+    "unidist.config",
+):
     try:
         __import__(mod_name)
     except ImportError:
@@ -52,6 +61,16 @@ def noop_decorator(*args, **kwargs):
     sys.modules["pyhdk"].__version__ = "999"
 if not hasattr(sys.modules["xgboost"], "Booster"):
     sys.modules["xgboost"].Booster = type("Booster", (object,), {})
+if not hasattr(sys.modules["unidist"], "remote"):
+    sys.modules["unidist"].remote = noop_decorator
+if not hasattr(sys.modules["unidist"], "core"):
+    sys.modules["unidist"].core = type("core", (object,), {})
+if not hasattr(sys.modules["unidist"].core, "base"):
+    sys.modules["unidist"].core.base = type("base", (object,), {})
+if not hasattr(sys.modules["unidist"].core.base, "object_ref"):
+    sys.modules["unidist"].core.base.object_ref = type("object_ref", (object,), {})
+if not hasattr(sys.modules["unidist"].core.base.object_ref, "ObjectRef"):
+    sys.modules["unidist"].core.base.object_ref.ObjectRef = type("ObjectRef", (object,), {})
 
 sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), "..")))
 import modin

docs/development/architecture.rst

@@ -216,8 +216,8 @@ documentation page on :doc:`contributing </development/contributing>`.
     - Uses the `Dask Futures`_ execution framework.
     - The storage format is `pandas` and the in-memory partition type is a pandas DataFrame.
     - For more information on the execution path, see the :doc:`pandas on Dask </flow/modin/core/execution/dask/implementations/pandas_on_dask/index>` page.
-- :doc:`pandas on Unidist </development/using_pandas_on_unidist>`
-    - Uses the Unidist_ execution framework.
+- :doc:`pandas on MPI </development/using_pandas_on_mpi>`
+    - Uses MPI_ through the Unidist_ execution framework.
     - The storage format is `pandas` and the in-memory partition type is a pandas DataFrame.
     - For more information on the execution path, see the :doc:`pandas on Unidist </flow/modin/core/execution/unidist/implementations/pandas_on_unidist/index>` page.
 - :doc:`pandas on Python </development/using_pandas_on_python>`
@@ -228,8 +228,8 @@ documentation page on :doc:`contributing </development/contributing>`.
     - Uses the Ray_ execution framework.
     - The storage format is `pandas` and the in-memory partition type is a pandas DataFrame.
     - For more information on the execution path, see the :doc:`experimental pandas on Ray </flow/modin/experimental/core/execution/ray/implementations/pandas_on_ray/index>` page.
-- pandas on Unidist (experimental)
-    - Uses the Unidist_ execution framework.
+- pandas on MPI (experimental)
+    - Uses MPI_ through the Unidist_ execution framework.
     - The storage format is `pandas` and the in-memory partition type is a pandas DataFrame.
     - For more information on the execution path, see the :doc:`experimental pandas on Unidist </flow/modin/experimental/core/execution/unidist/implementations/pandas_on_unidist/index>` page.
 - pandas on Dask (experimental)
@@ -375,6 +375,7 @@ details. The documentation covers most modules, with more docs being added every
 .. _Arrow tables: https://arrow.apache.org/docs/python/generated/pyarrow.Table.html
 .. _Ray: https://github.com/ray-project/ray
 .. _Unidist: https://github.com/modin-project/unidist
+.. _MPI: https://www.mpi-forum.org/
 .. _code: https://github.com/modin-project/modin/blob/master/modin/core/dataframe
 .. _Dask: https://github.com/dask/dask
 .. _Dask Futures: https://docs.dask.org/en/latest/futures.html

docs/development/index.rst

@@ -10,7 +10,7 @@ Development
     using_pandas_on_ray
     using_pandas_on_dask
     using_pandas_on_python
-    using_pandas_on_unidist
+    using_pandas_on_mpi
     using_hdk
     using_pyarrow_on_ray
 

docs/development/partition_api.rst

@@ -36,7 +36,7 @@ in the worker process that processes a function (please, refer to `Dask document
 
 Unidist engine
 --------------
-Currently, Modin only supports unidist on MPI backend. There is no mentioned above issue for
+Currently, Modin only supports MPI through unidist. There is no mentioned above issue for
 Modin on ``Unidist`` engine using ``MPI`` backend with ``pandas`` in-memory format
 because ``Unidist`` saves any objects in the MPI worker process that processes a function
 (please, refer to `Unidist documentation`_ for more information).

docs/development/using_pandas_on_unidist.rst docs/development/using_pandas_on_mpi.rst

@@ -1,14 +1,14 @@
-pandas on Unidist
-=================
+pandas on MPI through unidist
+=============================
 
-This section describes usage related documents for the pandas on Unidist component of Modin.
+This section describes usage related documents for the pandas on MPI through unidist component of Modin.
 
 Modin uses pandas as a primary memory format of the underlying partitions and optimizes queries
 ingested from the API layer in a specific way to this format. Thus, there is no need to care of choosing it
 but you can explicitly specify it anyway as shown below.
 
-One of the execution engines that Modin uses is Unidist. Currently, Modin only supports Unidist on MPI backend.
-To enable the pandas on Unidist execution using MPI backend you should set the following environment variables:
+One of the execution engines that Modin uses is MPI through unidist.
+To enable the pandas on MPI through unidist execution you should set the following environment variables:
 
 .. code-block:: bash
 
@@ -36,4 +36,21 @@ To run a python application you should use ``mpiexec -n 1 python <script.py>`` c
 For more information on how to run a python application with unidist on MPI backend
 please refer to `Unidist on MPI`_ section of the unidist documentation.
 
+As of unidist 0.5.0 there is support for a shared object store for MPI backend.
+The feature allows to improve performance in the workloads,
+where workers use same data multiple times by reducing data copies.
+You can enable the feature by setting the following environment variable:
+
+.. code-block:: bash
+
+   export UNIDIST_MPI_SHARED_OBJECT_STORE=True
+
+or turn it on in source code:
+
+.. code-block:: python
+
+   import unidist.config as unidist_cfg
+
+   unidist_cfg.MpiSharedObjectStore.put(True)
+
 .. _`Unidist on MPI`: https://unidist.readthedocs.io/en/latest/using_unidist/unidist_on_mpi.html

docs/flow/modin/core/execution/unidist/implementations/pandas_on_unidist/index.rst

@@ -6,7 +6,8 @@ PandasOnUnidist Execution
 Queries that perform data transformation, data ingress or data egress using the `pandas on Unidist` execution
 pass through the Modin components detailed below.
 
-To enable `pandas on Unidist` execution, please refer to the usage section in :doc:`pandas on Unidist </development/using_pandas_on_unidist>`.
+To enable `pandas on MPI through unidist` execution,
+please refer to the usage section in :doc:`pandas on MPI through unidist </development/using_pandas_on_mpi>`.
 
 Data Transformation
 '''''''''''''''''''

docs/getting_started/faq.rst

@@ -119,7 +119,7 @@ and Modin will do computation with that engine:
     pip install "modin[dask]" # Install Modin dependencies and Dask to run on Dask
     export MODIN_ENGINE=dask  # Modin will use Dask
 
-    pip install "modin[unidist]" # Install Modin dependencies and Unidist to run on Unidist.
+    pip install "modin[mpi]" # Install Modin dependencies and MPI to run on MPI through unidist.
     export MODIN_ENGINE=unidist  # Modin will use Unidist
     export UNIDIST_BACKEND=mpi   # Unidist will use MPI backend.