Scipy, qutip, pandas problem

[martenrichter]

I have a bit weird dependency problem.
I have the following environment setup:

name: xeus-python-kernel
channels:
  - https://repo.mamba.pm/emscripten-forge
  - conda-forge
dependencies:
  - python=3.11
  - xeus-python
  - ipycanvas
  - ipywidgets
  - matplotlib
  - plotly
  - pandas
  - nbformat
  - numpy
  - anywidget
  - scipy
  - sympy
  - qutip

If I build it without qutip, everything works fine.
If I build it with qutip, loading

import plotly.express as px

fails, as it can not load pandas.

If I only do

import pandas

it fails with

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[4], line 1
----> 1 import pandas

File /lib/python3.11/site-packages/pandas/__init__.py:49
     46 # let init-time option registration happen
     47 import pandas.core.config_init  # pyright: ignore[reportUnusedImport] # noqa: F401
---> 49 from pandas.core.api import (
     50     # dtype
     51     ArrowDtype,
     52     Int8Dtype,
     53     Int16Dtype,
     54     Int32Dtype,
     55     Int64Dtype,
     56     UInt8Dtype,
     57     UInt16Dtype,
     58     UInt32Dtype,
     59     UInt64Dtype,
     60     Float32Dtype,
     61     Float64Dtype,
     62     CategoricalDtype,
     63     PeriodDtype,
     64     IntervalDtype,
     65     DatetimeTZDtype,
     66     StringDtype,
     67     BooleanDtype,
     68     # missing
     69     NA,
     70     isna,
     71     isnull,
     72     notna,
     73     notnull,
     74     # indexes
     75     Index,
     76     CategoricalIndex,
     77     RangeIndex,
     78     MultiIndex,
     79     IntervalIndex,
     80     TimedeltaIndex,
     81     DatetimeIndex,
     82     PeriodIndex,
     83     IndexSlice,
     84     # tseries
     85     NaT,
     86     Period,
     87     period_range,
     88     Timedelta,
     89     timedelta_range,
     90     Timestamp,
     91     date_range,
     92     bdate_range,
     93     Interval,
     94     interval_range,
     95     DateOffset,
     96     # conversion
     97     to_numeric,
     98     to_datetime,
     99     to_timedelta,
    100     # misc
    101     Flags,
    102     Grouper,
    103     factorize,
    104     unique,
    105     value_counts,
    106     NamedAgg,
    107     array,
    108     Categorical,
    109     set_eng_float_format,
    110     Series,
    111     DataFrame,
    112 )
    114 from pandas.core.dtypes.dtypes import SparseDtype
    116 from pandas.tseries.api import infer_freq

File /lib/python3.11/site-packages/pandas/core/api.py:1
----> 1 from pandas._libs import (
      2     NaT,
      3     Period,
      4     Timedelta,
      5     Timestamp,
      6 )
      7 from pandas._libs.missing import NA
      9 from pandas.core.dtypes.dtypes import (
     10     ArrowDtype,
     11     CategoricalDtype,
   (...)
     14     PeriodDtype,
     15 )

File /lib/python3.11/site-packages/pandas/_libs/__init__.py:18
     16 import pandas._libs.pandas_parser  # isort: skip # type: ignore[reportUnusedImport]
     17 import pandas._libs.pandas_datetime  # noqa: F401 # isort: skip # type: ignore[reportUnusedImport]
---> 18 from pandas._libs.interval import Interval
     19 from pandas._libs.tslibs import (
     20     NaT,
     21     NaTType,
   (...)
     26     iNaT,
     27 )

File interval.pyx:1, in init pandas._libs.interval()

File hashtable.pyx:1, in init pandas._libs.hashtable()

File missing.pyx:1, in init pandas._libs.missing()

File /lib/python3.11/site-packages/pandas/_libs/tslibs/__init__.py:40
      1 __all__ = [
      2     "dtypes",
      3     "localize_pydatetime",
   (...)
     36     "is_supported_dtype",
     37 ]
     39 from pandas._libs.tslibs import dtypes  # pylint: disable=import-self
---> 40 from pandas._libs.tslibs.conversion import localize_pydatetime
     41 from pandas._libs.tslibs.dtypes import (
     42     Resolution,
     43     periods_per_day,
     44     periods_per_second,
     45 )
     46 from pandas._libs.tslibs.nattype import (
     47     NaT,
     48     NaTType,
     49     iNaT,
     50     nat_strings,
     51 )

File conversion.pyx:1, in init pandas._libs.tslibs.conversion()

File qutip/core/data/base.pyx:6, in init qutip.core.data.base()

File /lib/python3.11/site-packages/qutip/__init__.py:33
     26     del matplotlib
     29 # -----------------------------------------------------------------------------
     30 # Load modules
     31 #
---> 33 from .core import *
     34 from .solver import *
     35 from .solver import nonmarkov

File /lib/python3.11/site-packages/qutip/core/__init__.py:2
      1 from .options import *
----> 2 from .coefficient import *
      3 from .qobj import *
      4 from .cy.qobjevo import *

File /lib/python3.11/site-packages/qutip/core/coefficient.py:3
      1 import numpy as np
      2 import scipy
----> 3 import scipy.interpolate
      4 import os
      5 import sys

File /lib/python3.11/site-packages/scipy/interpolate/__init__.py:167
      1 """
      2 ========================================
      3 Interpolation (:mod:`scipy.interpolate`)
   (...)
    165 (should not be used in new code).
    166 """
--> 167 from ._interpolate import *
    168 from ._fitpack_py import *
    170 # New interface to fitpack library:

File /lib/python3.11/site-packages/scipy/interpolate/_interpolate.py:16
     14 from ._polyint import _Interpolator1D
     15 from . import _ppoly
---> 16 from .interpnd import _ndim_coords_from_arrays
     17 from ._bsplines import make_interp_spline, BSpline
     20 def lagrange(x, w):

File interpnd.pyx:1, in init scipy.interpolate.interpnd()

AttributeError: module 'scipy.spatial._qhull' has no attribute '__pyx_capi__'

so it seems that pandas invokes qutip (Why?), which invokes in turn scipy.interpolate. In this causes the other packages to fail.
Running

import scipy.interpolate

confirms it.

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[2], line 1
----> 1 import scipy.interpolate

File /lib/python3.11/site-packages/scipy/interpolate/__init__.py:167
      1 """
      2 ========================================
      3 Interpolation (:mod:`scipy.interpolate`)
   (...)
    165 (should not be used in new code).
    166 """
--> 167 from ._interpolate import *
    168 from ._fitpack_py import *
    170 # New interface to fitpack library:

File /lib/python3.11/site-packages/scipy/interpolate/_interpolate.py:16
     14 from ._polyint import _Interpolator1D
     15 from . import _ppoly
---> 16 from .interpnd import _ndim_coords_from_arrays
     17 from ._bsplines import make_interp_spline, BSpline
     20 def lagrange(x, w):

File interpnd.pyx:1, in init scipy.interpolate.interpnd()

AttributeError: module 'scipy.spatial._qhull' has no attribute '__pyx_capi__'

Of course, scipy.interpolate is deprecated.
The questions is, why it is failing?
Are the deprecated functions missing in current scipy emscripten build and is this causing the problem? Or is a change upstream in qutip required?

[martenrichter]

Linking
jupyterlite/xeus-lite-demo#46
which is probably related. But I think that this is a packaging issue.

[martinRenou]

@DerThorsten have you seen this?

AttributeError: module 'scipy.spatial._qhull' has no attribute '__pyx_capi__'

I have a feeling I've already seen this message before, I don't remember what we did about it or if we even fixed it.

[DerThorsten]

scipy.spatial._qhull was problematic before...as anything scipy related.
but I dont have a quick solution

[martenrichter]

With pyodide it works, I think they have a newer version?
If I understood it correctly emscripten-forge is using an pyodide build, correct?

[martenrichter]

Old fortran code seems to be one main problem. I just googled a bit, did you try https://github.com/r-wasm/flang-wasm?tab=readme-ov-file for building scipy?

[martenrichter]

Ok, I am now trying to look a bit deeper into the problem.
I have put debug logs at the emscripten function for looking up files on the file system.
The funny thing is even, if the message says in the end:
AttributeError: module 'scipy.spatial._qhull' has no attribute '__pyx_capi__'
I can not see that the code tries to access spatial or qhull, all that I see is:

look for path /lib/python3.11/site-packages/scipy
xpython.js:8 path found /lib/python3.11/site-packages/scipy
xpython.js:8 look for path /lib/python3.11/site-packages/scipy/interpolate/__init__.cpython-311-wasm32-emscripten.so
xpython.js:8 path not found error /lib/python3.11/site-packages/scipy/interpolate/__init__.cpython-311-wasm32-emscripten.so
xpython.js:8 look for path /lib/python3.11/site-packages/scipy/interpolate/__init__.abi3.so
xpython.js:8 path not found error /lib/python3.11/site-packages/scipy/interpolate/__init__.abi3.so
xpython.js:8 look for path /lib/python3.11/site-packages/scipy/interpolate/__init__.so
xpython.js:8 path not found error /lib/python3.11/site-packages/scipy/interpolate/__init__.so
xpython.js:8 look for path /lib/python3.11/site-packages/scipy/interpolate/__init__.py
xpython.js:8 path found /lib/python3.11/site-packages/scipy/interpolate/__init__.py
xpython.js:8 look for path /lib/python3.11/site-packages/scipy/interpolate/__init__.py
xpython.js:8 path found /lib/python3.11/site-packages/scipy/interpolate/__init__.py
xpython.js:8 open /lib/python3.11/site-packages/scipy/interpolate/__pycache__/__init__.cpython-311.pyc
xpython.js:8 look for path /lib/python3.11/site-packages/scipy/interpolate/__pycache__/__init__.cpython-311.pyc
xpython.js:8 path found /lib/python3.11/site-packages/scipy/interpolate/__pycache__/__init__.cpython-311.pyc
xpython.js:8 look for path /lib/python3.11/site-packages/scipy/interpolate/__pycache__/__init__.cpython-311.pyc
xpython.js:8 path found /lib/python3.11/site-packages/scipy/interpolate/__pycache__/__init__.cpython-311.pyc
xpython.js:8 look for path /lib/python3.11/site-packages/scipy/interpolate
xpython.js:8 path found /lib/python3.11/site-packages/scipy/interpolate
xpython.js:8 look for path /lib/python3.11/site-packages/scipy/interpolate/_interpolate.py
xpython.js:8 path found /lib/python3.11/site-packages/scipy/interpolate/_interpolate.py
xpython.js:8 look for path /lib/python3.11/site-packages/scipy/interpolate/_interpolate.py
xpython.js:8 path found /lib/python3.11/site-packages/scipy/interpolate/_interpolate.py
xpython.js:8 open /lib/python3.11/site-packages/scipy/interpolate/__pycache__/_interpolate.cpython-311.pyc
xpython.js:8 look for path /lib/python3.11/site-packages/scipy/interpolate/__pycache__/_interpolate.cpython-311.pyc
xpython.js:8 path found /lib/python3.11/site-packages/scipy/interpolate/__pycache__/_interpolate.cpython-311.pyc
xpython.js:8 look for path /lib/python3.11/site-packages/scipy/interpolate/__pycache__/_interpolate.cpython-311.pyc
xpython.js:8 path found /lib/python3.11/site-packages/scipy/interpolate/__pycache__/_interpolate.cpython-311.pyc
xpython.js:8 look for path /lib/python3.11/site-packages/scipy/interpolate
xpython.js:8 path found /lib/python3.11/site-packages/scipy/interpolate
xpython.js:8 look for path /lib/python3.11/site-packages/scipy/interpolate/interpnd.cpython-311-wasm32-emscripten.so
xpython.js:8 path found /lib/python3.11/site-packages/scipy/interpolate/interpnd.cpython-311-wasm32-emscripten.so
xpython.js:8 look for path /lib/python3.11/site-packages/IPython/core/interactiveshell.py
xpython.js:8 path found /lib/python3.11/site-packages/IPython/core/interactiveshell.py
xpython.js:8 look for path /tmp/xpython_42/245123379.py
xpython.js:8 path not found error /tmp/xpython_42/245123379.py
xpython.js:8 open /tmp/xpython_42/245123379.py
xpython.js:8 can not open fs error 44 /tmp/xpython_42/245123379.py
xpython.js:8 look for path /lib/python3.11/site-packages/scipy/interpolate/__init__.py
xpython.js:8 path found /lib/python3.11/site-packages/scipy/interpolate/__init__.py
xpython.js:8 look for path /lib/python3.11/site-packages/scipy/interpolate/_interpolate.py
xpython.js:8 path found /lib/python3.11/site-packages/scipy/interpolate/_interpolate.py
xpython.js:8 look for path interpnd.pyx
xpython.js:8 path not found error interpnd.pyx
xpython.js:8 look for path /lib/python3.11/site-packages/scipy/interpolate/__init__.py
xpython.js:8 path found /lib/python3.11/site-packages/scipy/interpolate/__init__.py
xpython.js:8 look for path /lib/python3.11/site-packages/scipy/interpolate/_interpolate.py
xpython.js:8 path found /lib/python3.11/site-packages/scipy/interpolate/_interpolate.py
xpython.js:8 look for path ./interpnd.pyx
xpython.js:8 path not found error ./interpnd.pyx
xpython.js:8 look for path interpnd.pyx
xpython.js:8 path not found error interpnd.pyx
xpython.js:8 look for path /lib/python3.11/site-packages/interpnd.pyx
xpython.js:8 path not found error /lib/python3.11/site-packages/interpnd.pyx
xpython.js:8 look for path /usr/lib/python3.11/interpnd.pyx
xpython.js:8 path not found error /usr/lib/python3.11/interpnd.pyx
xpython.js:8 look for path /lib/python311.zip/interpnd.pyx
xpython.js:8 path not found error /lib/python311.zip/interpnd.pyx
xpython.js:8 look for path /lib/python3.11/interpnd.pyx
xpython.js:8 path not found error /lib/python3.11/interpnd.pyx
xpython.js:8 look for path /lib/python3.11/lib-dynload/interpnd.pyx
xpython.js:8 path not found error /lib/python3.11/lib-dynload/interpnd.pyx

I have put break points at filesystem stat and open, and that the file interpnd.pyx is not found seems to be the problem. And it is definitely not part of the generated tar.gz.

[martenrichter]

I am wondering how loading pyx work with jupyter lite xeus? Is cython running inside emscripten, or should a mechanism exist, that replaces pyx with cpython-311-wasm32-emscripten.so, which I have see as the only extension for other package loading.
If so, can someone give me a pointer to where this should occur in the kernel (I mean, which package?)? (The calling code is all wasm and not annotated, and tracking the assembly is a bit pointless.)