Failing CI nbval output on single notebook

[fmigneault]

The DACCS Jenkins CI hosted on CRIM is functional once again as seen here (bird-house/birdhouse-deploy#493 (comment)) and on nightly runs, as shown below.

Everything succeeds, except the following test:
https://github.com/Ouranosinc/PAVICS-landing/blob/master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb

19:08:59  ============================= test session starts ==============================
19:08:59  platform linux -- Python 3.11.10, pytest-8.3.3, pluggy-1.5.0
19:08:59  rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
19:08:59  plugins: anyio-4.6.2.post1, dash-2.18.1, nbval-0.11.0, tornasync-0.6.0.post2, xdist-3.6.1
19:08:59  collected 302 items
19:08:59  
19:09:07  notebooks-auth/geoserver.ipynb ..................                        [  5%]
19:10:18  notebooks-auth/test_cowbird_jupyter.ipynb ..........                     [  9%]
19:10:18  notebooks-auth/test_thredds.ipynb ...........                            [ 12%]
19:11:49  pavics-sdi-master/docs/source/notebooks/CaSR_basic.ipynb ......          [ 14%]
19:12:04  pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [ 17%]
19:12:10  pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 19%]
19:21:37  pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 23%]
19:21:37  pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 23%]
19:21:37  ...............                                                          [ 28%]
19:21:42  pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 30%]
19:21:50  pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb .......           [ 32%]
19:22:04  pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 34%]
19:22:11  pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 36%]
19:22:15  pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 38%]
19:25:43  pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 43%]
19:26:53  .............                                                            [ 47%]
19:26:55  pavics-sdi-master/docs/source/notebooks/rendering.ipynb .....            [ 49%]
19:26:57  pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 51%]
19:27:18  .................                                                        [ 57%]
19:27:26  pavics-sdi-master/docs/source/notebooks/subsetting.ipynb ......          [ 59%]
19:27:27  pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 59%]
19:27:49  .........                                                                [ 62%]
19:27:58  finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 66%]
19:28:07  finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 68%]
19:28:08  PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 68%]
19:28:12  .....                                                                    [ 70%]
19:29:20  PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 70%]
19:29:34  ............                                                             [ 74%]
19:29:56  PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 75%]
19:30:43  .....s.                                                                  [ 77%]
19:30:49  PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 77%]
19:30:58  ..                                                                       [ 78%]
19:31:20  PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 78%]
19:32:16  F........                                                                [ 81%]
19:32:38  PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-6Regridding_Conversion.ipynb . [ 82%]
19:38:52  ....                                                                     [ 83%]
19:38:52  PAVICS-landing-master/content/notebooks/hydrology/PAVICStutorial_Hydrology-01_Intro.ipynb . [ 83%]
19:38:54  ....                                                                     [ 85%]
19:38:58  PAVICS-landing-master/content/notebooks/hydrology/PAVICStutorial_Hydrology-02_Calibration.ipynb . [ 85%]
19:39:06  .....                                                                    [ 87%]
19:39:10  PAVICS-landing-master/content/notebooks/hydrology/PAVICStutorial_Hydrology-03_Watershed_properties.ipynb . [ 87%]
19:39:29  .............                                                            [ 91%]
19:39:33  PAVICS-landing-master/content/notebooks/hydrology/PAVICStutorial_Hydrology-04_Time_series_analysis.ipynb . [ 92%]
19:39:35  ......                                                                   [ 94%]
19:39:37  notebooks/hummingbird.ipynb ............                                 [ 98%]
19:42:11  notebooks/stress-tests.ipynb ......                                      [100%]
19:42:11  
19:42:11  =================================== FAILURES ===================================
19:42:11  _ PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb::Cell 1 _
19:42:11  Notebook cell execution failed
19:42:11  Cell 1: Cell outputs differ
19:42:11  
19:42:11  Input:
19:42:11  ##
19:42:11  bokeh.io.output_notebook()
19:42:11  from bokeh.resources import settings
19:42:11  
19:42:11  settings.resources = "inline"
19:42:11  
19:42:11  logging.getLogger().disabled = True
19:42:11  warnings.simplefilter("ignore")
19:42:11  pn.extension()
19:42:11  
19:42:11  
19:42:11  output = "/notebook_dir/writable-workspace/tmp/tutorial3/output"
19:42:11  infolder = Path(output)
19:42:11  ds_all = []
19:42:11  for scen in ["ssp245", "ssp370"]:
19:42:11      ds_vars = []
19:42:11      for v in ["tx_mean", "tx_days_above_27"]:
19:42:11          for f in ["seasonal", "annual"]:
19:42:11              # create a list of datasets for which the time dimension has been "unstacked"
19:42:11              ncfiles = [
19:42:11                  xs.utils.unstack_dates(xr.open_dataset(d, decode_timedelta=False))
19:42:11                  for d in infolder.glob(f"{v}*{f}*{scen}*.nc")
19:42:11              ]
19:42:11  
19:42:11              # Create an ensemble dataset from the simulations for the given SSP
19:42:11              # Add a new scen dimension and coordinate
19:42:11              ds_vars.append(
19:42:11                  xens.create_ensemble(ncfiles, calendar="standard").expand_dims(
19:42:11                      scen=[scen]
19:42:11                  )
19:42:11              )
19:42:11      # Create a single dataset with both variables
19:42:11      ds_all.append(xr.merge(ds_vars))
19:42:11  
19:42:11  # concatenate the 2 SSP datasets together along the new 'scen' dim
19:42:11  ds_ens = xr.concat(ds_all, dim="scen")
19:42:11  ds_ens = ds_ens.sel(time=slice("1981", "2100"))  # select years 1971-2100
19:42:11  
19:42:11  # It is usually more intuitive to visualize temperature in Celsius
19:42:11  for v in ds_ens.data_vars:
19:42:11      if units.units2pint(ds_ens[v]) == "kelvin":
19:42:11          ds_ens[v] = units.convert_units_to(ds_ens[v], "degC")
19:42:11  
19:42:11  # rearrange season order to something more logical
19:42:11  ds_ens = ds_ens.sel(season=["annual", "DJF", "MAM", "JJA", "SON"])
19:42:11  
19:42:11  ds_ens
19:42:11  
19:42:11  Traceback:
19:42:11   mismatch 'text/plain'
19:42:11  
19:42:11   assert reference_output == test_output failed:
19:42:11  
19:42:11    '<xarray.Data...    EPSG:4326' == '<xarray.Data...    EPSG:4326'
19:42:11    
19:42:11    Skipping 992 identical leading characters in diff, use -v to show
19:42:11    - utes: (12/79)
19:42:11    ?           ^^
19:42:11    + utes: (12/80)
19:42:11    ?           ^^
19:42:11          Conventions:                     CF-1.7 CMIP-6.2
19:42:11          Notes:                           Regridded on the grid of RDRS v2.1, then...
19:42:11          activity_id:                     CMIP
19:42:11          branch_method:                   standard
19:42:11          branch_time_in_child:            0.0
19:42:11    -     branch_time_in_parent:           21915.0
19:42:11    ?                                      ^^^^^
19:42:11    +     branch_time_in_parent:           0.0
19:42:11    ?                                      ^
19:42:11          ...                              ...
19:42:11          license_type:                    permissive
19:42:11          terms_of_use:                    In addition to the provided licence, the...
19:42:11          attribution:                     Use of this dataset should be acknowledg...
19:42:11          modeling_realm:                  atmos
19:42:11    -     source_institution:              CSIRO
19:42:11    ?                                      ^^ ^^
19:42:11    +     source_institution:              MPI-M
19:42:11    ?                                      ^^ ^^
19:42:11          crs:                             EPSG:4326
19:42:11 

[tlvu]

I am completely stumped by this one because running the exact same notebook, hitting exactly the same server, inside an exactly same runtime environment (the Jupyter docker image) do not produce this error on our side !!!

[fmigneault]

Yes. I'm not sure either what is happening, but it is doing it consistently for each PR/commit since the CI is back in operation.

[tlvu]

Took some time but finally I am able to reproduce this one.

It seems to only happen on Ubuntu 24.04 with docker engine from Docker and not from Ubuntu (Ubuntu also provide its own docker engine package (apt install docker.io)). Rocky8 and 9, not able to reproduce either.

Anyways, here is the slightly different error I got on my side. It's consistent. I think this has maybe something to do with sorting the datasets after the merge. Will attempt some blind fixes tomorrow, or raise the issue with my colleagues.

22:29:38  _ PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb::Cell 1 _
22:29:38  Notebook cell execution failed
22:29:38  Cell 1: Cell outputs differ
22:29:38  
22:29:38  Input:
22:29:38  ##
22:29:38  bokeh.io.output_notebook()
22:29:38  from bokeh.resources import settings
22:29:38  
22:29:38  settings.resources = "inline"
22:29:38  
22:29:38  logging.getLogger().disabled = True
22:29:38  warnings.simplefilter("ignore")
22:29:38  pn.extension()
22:29:38  
22:29:38  
22:29:38  output = "/notebook_dir/writable-workspace/tmp/tutorial3/output"
22:29:38  infolder = Path(output)
22:29:38  ds_all = []
22:29:38  for scen in ["ssp245", "ssp370"]:
22:29:38      ds_vars = []
22:29:38      for v in ["tx_mean", "tx_days_above_27"]:
22:29:38          for f in ["seasonal", "annual"]:
22:29:38              # create a list of datasets for which the time dimension has been "unstacked"
22:29:38              ncfiles = [
22:29:38                  xs.utils.unstack_dates(xr.open_dataset(d, decode_timedelta=False))
22:29:38                  for d in infolder.glob(f"{v}*{f}*{scen}*.nc")
22:29:38              ]
22:29:38  
22:29:38              # Create an ensemble dataset from the simulations for the given SSP
22:29:38              # Add a new scen dimension and coordinate
22:29:38              ds_vars.append(
22:29:38                  xens.create_ensemble(ncfiles, calendar="standard").expand_dims(
22:29:38                      scen=[scen]
22:29:38                  )
22:29:38              )
22:29:38      # Create a single dataset with both variables
22:29:38      ds_all.append(xr.merge(ds_vars))
22:29:38  
22:29:38  # concatenate the 2 SSP datasets together along the new 'scen' dim
22:29:38  ds_ens = xr.concat(ds_all, dim="scen")
22:29:38  ds_ens = ds_ens.sel(time=slice("1981", "2100"))  # select years 1971-2100
22:29:38  
22:29:38  # It is usually more intuitive to visualize temperature in Celsius
22:29:38  for v in ds_ens.data_vars:
22:29:38      if units.units2pint(ds_ens[v]) == "kelvin":
22:29:38          ds_ens[v] = units.convert_units_to(ds_ens[v], "degC")
22:29:38  
22:29:38  # rearrange season order to something more logical
22:29:38  ds_ens = ds_ens.sel(season=["annual", "DJF", "MAM", "JJA", "SON"])
22:29:38  
22:29:38  ds_ens
22:29:38  
22:29:38  Traceback:
22:29:38   mismatch 'text/plain'
22:29:38  
22:29:38   assert reference_output == test_output failed:
22:29:38  
22:29:38    '<xarray.Data...    EPSG:4326' == '<xarray.Data...    EPSG:4326'
22:29:38    
22:29:38    Skipping 1296 identical leading characters in diff, use -v to show
22:29:38    -           90885.0
22:29:38    ?           - ---
22:29:38    +           0.0
22:29:38          ...                              ...
22:29:38          license_type:                    permissive
22:29:38          terms_of_use:                    In addition to the provided licence, the...
22:29:38          attribution:                     Use of this dataset should be acknowledg...
22:29:38          modeling_realm:                  atmos
22:29:38    -     source_institution:              INM
22:29:38    ?                                       ^
22:29:38    +     source_institution:              MPI-M
22:29:38    ?                                      ++ ^
22:29:38          crs:                             EPSG:4326
22:29:38