Add optional default config to disable some notebooks in Jenkins

[tlvu]

To be used together with Ouranosinc/jenkins-config#17.

If Jenkins env.local contains export DEFAULT_CONFIG_PARAMETERS_SCRIPT_URL="https://raw.githubusercontent.com/Ouranosinc/PAVICS-e2e-workflow-tests/refs/heads/master/test-override/jenkins-params-default.include.sh" notebooks that are known to have problems will be skipped so that Jenkins do not fail for known reasons.

This is related to bird-house/birdhouse-deploy#496. If raven notebooks are enabled by default, then we need to blacklist one of them because that one never passed, see test-override/jenkins-params-default.include.sh (the file that Jenkins will be pointed to). The raven notebook is finally disabled by this PR CSHS-CWRA/RavenPy#460.

In this same effort, we also add the ability to blacklist notebooks only at the "generating output.ipynb" stage, but not in "pytest stage". The reason is that those notebooks have some cells that are disabled during the "pytest stage" because they take too long so then there are no reason to generate the output because 1) it will timeout anyways, 2) the output won't match since some cells are disabled during "pytest stage".

@fmigneault
CRIM should simply point to our https://raw.githubusercontent.com/Ouranosinc/PAVICS-e2e-workflow-tests/refs/heads/master/test-override/jenkins-params-default.include.sh file so we will be responsible for keeping up-to-date this list of known failures to prevent breaking CRIM pipeline.

[fmigneault]

Is there some pytest-native --skip / --ignore or similar parameter that can be leveraged?

Can't the NBVAL OUTPUT ignore be used for them?

If those cannot work, can't we use the other skips already in place, such as moving them into https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/tree/master/notebooks-notworking or an equivalent in their corresponding repositories.

Can you provide more details about the failing notebooks? Why are they failing? If they fail for other reasons, why keep them (they don't seem like good tutorials to showcase if they don't work)?

I would really rather avoid introducing yet another custom script. There's already too many dynamic loads/enable/disable in that pipeline, that it takes days to investigate any minor issue.

[tlvu]

Is there some pytest-native --skip / --ignore or similar parameter that can be leveraged?

There is --ignore but I'd rather not hardcode but using the plugin mechanism anyways. So instead of removing from the list of $NOTEBOOKS, I can use --ignore, you prefer this?

Can't the NBVAL OUTPUT ignore be used for them?

No because it's not the output, it's the code that fail.

If those cannot work, can't we use the other skips already in place, such as moving them into https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/tree/master/notebooks-notworking or an equivalent in their corresponding repositories.

The notebook is in another repo (RavenPy), not in PAVICS-e2e-workflow-tests so that folder notebooks-notworking is not there.

Can you provide more details about the failing notebooks? Why are they failing? If they fail for other reasons, why keep them (they don't seem like good tutorials to showcase if they don't work)?

The authentication password on ESGF do not work anymore. For those with active password, I believe the notebook should still work. But good point, I'll check with David, he used to provide me with up-to-date password. But that means CRIM should also get your own password since it's David personal password.

I would really rather avoid introducing yet another custom script. There's already too many dynamic loads/enable/disable in that pipeline, that it takes days to investigate any minor issue.

Understood, you don't have to enable it if you do not feel confortable. We can find a compromise. I'll see if I can disable that RavenPy nb by another mean. I still have to provide an example script to show how the new default override can be used.

For us, we will also leverage this mechanism to disable some more notebooks in the "generate output using jupyter nbconvert" phase. Your pipeline do not enable that phase so you do not need it.

Just to be clear I actually did not introduce any new mechanisms. It's the existing mechanism to dynamically load a config that override the many Jenkins parameter that I move out to a function so I can call it twice: once for a default override (new) and once for the interactive from the Jenkins build request (existing).

In order to make Jenkins config not specific to Ouranos, these dynamic override are more flexible than adding more params to Jenkins. These dynamic override can start testing repos unknown to the current Jenkins config.

[tlvu]

Discussed with David, we will simply disable that notebook in CSHS-CWRA/RavenPy#460.

So this means you guys so not need to use this new feature anymore. But it'll be ready when you needed.

Can I merge this PR and the corresponding Ouranosinc/jenkins-config#17? It's 100% backward compatible, nothing to do on your side if you do not want to enable it.

[fmigneault]

There is --ignore but I'd rather not hardcode but using the plugin mechanism anyways. So instead of removing from the list of $NOTEBOOKS, I can use --ignore, you prefer this?

I'm not sure to see the different between some kind of TEST_IGNORE_NOTEBOOKS variable that would do the filtering job before pytest nbval call, and the explicitly hardcoded paths in enable_resulting_nb() that are toggled via DEFAULT_CONFIG_OVERRIDE_SCRIPT_URL?

Actually, wouldn't it be already possible via $PYTEST_EXTRA_OPTS if I passed --ignore <notebook_i_dont_like> in my env?

[tlvu]

Actually, wouldn't it be already possible via $PYTEST_EXTRA_OPTS if I passed --ignore <notebook_i_dont_like> in my env?

Yes that would work for the pytest step but not for the jupyter nbconvert step so that's why enable_resulting_nb() is used in the jupyter nbconvert step.

[tlvu]

@fmigneault Renamed var and point to real usage on jenkins-config side for clarify: e2c3d7e

[tlvu]

@fmigneault You okay with this PR now?

[fmigneault]

Thanks for the clarifications!

[fmigneault]

@tlvu
Yesterday (2025-02-26), we got our first nightly end-2-end run that passed entirely in a very long time. 🎉

To be noted that I did NOT apply any change mentioned in #142 (comment). It just ran with the usual references. Therefore, I have no idea why the notebooks listed in https://raw.githubusercontent.com/Ouranosinc/PAVICS-e2e-workflow-tests/refs/heads/master/test-override/jenkins-params-default.include.sh ran successfully this time. 🤔

http://daccs-jenkins.crim.ca/job/PAVICS-e2e-workflow-tests/job/master/54

19:08:21  ============================= test session starts ==============================
19:08:21  platform linux -- Python 3.11.10, pytest-8.3.3, pluggy-1.5.0
19:08:21  rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
19:08:21  plugins: anyio-4.6.2.post1, dash-2.18.1, nbval-0.11.0, tornasync-0.6.0.post2, xdist-3.6.1
19:08:21  collected 546 items
19:08:21  
19:08:31  notebooks-auth/geoserver.ipynb ..................                        [  3%]
19:09:44  notebooks-auth/test_cowbird_jupyter.ipynb ..........                     [  5%]
19:09:46  notebooks-auth/test_thredds.ipynb ...........                            [  7%]
19:11:32  pavics-sdi-master/docs/source/notebooks/CaSR_basic.ipynb ......          [  8%]
19:22:53  pavics-sdi-master/docs/source/notebooks/FAQ_dask_parallel.ipynb .s...... [  9%]
19:22:53                                                                           [  9%]
19:22:59  pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [ 10%]
19:23:06  pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 12%]
19:34:10  pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 14%]
19:34:10  pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 14%]
19:34:10  ...............                                                          [ 17%]
19:34:15  pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 18%]
19:37:25  pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb .......           [ 19%]
19:37:33  pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 20%]
19:37:40  pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 21%]
19:37:45  pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 22%]
19:41:39  pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 25%]
19:42:46  .............                                                            [ 27%]
19:42:48  pavics-sdi-master/docs/source/notebooks/rendering.ipynb .....            [ 28%]
19:42:49  pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 30%]
19:43:12  .................                                                        [ 33%]
19:43:19  pavics-sdi-master/docs/source/notebooks/subsetting.ipynb ......          [ 34%]
19:43:20  pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 34%]
19:43:37  .........                                                                [ 36%]
19:43:46  finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 38%]
19:43:55  finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 39%]
19:43:56  PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 39%]
19:44:01  .....                                                                    [ 40%]
19:45:22  PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 40%]
19:45:47  ............                                                             [ 42%]
19:46:55  PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 43%]
19:47:55  .....s.                                                                  [ 44%]
19:48:01  PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 44%]
19:48:10  ..                                                                       [ 44%]
19:48:32  PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 45%]
19:49:29  .........                                                                [ 46%]
19:49:51  PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-6Regridding_Conversion.ipynb . [ 46%]
19:57:13  ....                                                                     [ 47%]
19:57:13  PAVICS-landing-master/content/notebooks/hydrology/PAVICStutorial_Hydrology-01_Intro.ipynb . [ 47%]
19:57:19  ....                                                                     [ 48%]
19:57:23  PAVICS-landing-master/content/notebooks/hydrology/PAVICStutorial_Hydrology-02_Calibration.ipynb . [ 48%]
19:57:31  .....                                                                    [ 49%]
19:57:36  PAVICS-landing-master/content/notebooks/hydrology/PAVICStutorial_Hydrology-03_Watershed_properties.ipynb . [ 49%]
19:57:51  .............                                                            [ 52%]
19:57:57  PAVICS-landing-master/content/notebooks/hydrology/PAVICStutorial_Hydrology-04_Time_series_analysis.ipynb . [ 52%]
19:57:57  ......                                                                   [ 53%]
19:58:12  raven-main/docs/source/notebooks/Region_selection.ipynb .........        [ 55%]
19:58:14  raven-main/docs/source/notebooks/Subset_climate_data_over_watershed.ipynb . [ 55%]
19:58:40  ......                                                                   [ 56%]
19:58:42  RavenPy-master/docs/notebooks/00_Introduction_to_JupyterLab.ipynb ...... [ 57%]
19:58:42                                                                           [ 57%]
19:58:44  RavenPy-master/docs/notebooks/01_Getting_watershed_boundaries.ipynb .... [ 58%]
19:58:57  ....                                                                     [ 58%]
19:59:01  RavenPy-master/docs/notebooks/02_Extract_geographical_watershed_properties.ipynb . [ 59%]
19:59:07  .............                                                            [ 61%]
19:59:39  RavenPy-master/docs/notebooks/03_Extracting_forcing_data.ipynb ......... [ 63%]
20:00:47  ..                                                                       [ 63%]
20:00:50  RavenPy-master/docs/notebooks/04_Emulating_hydrological_models.ipynb ... [ 64%]
20:00:59  .................                                                        [ 67%]
20:01:05  RavenPy-master/docs/notebooks/05_Advanced_RavenPy_configuration.ipynb .. [ 67%]
20:01:15  ...........                                                              [ 69%]
20:01:26  RavenPy-master/docs/notebooks/06_Raven_calibration.ipynb ......          [ 70%]
20:01:30  RavenPy-master/docs/notebooks/07_Making_and_using_hotstart_files.ipynb . [ 70%]
20:01:36  .....                                                                    [ 71%]
20:01:37  RavenPy-master/docs/notebooks/08_Getting_and_bias_correcting_CMIP6_data.ipynb . [ 71%]
20:09:32  ................                                                         [ 74%]
20:09:35  RavenPy-master/docs/notebooks/09_Hydrological_impacts_of_climate_change.ipynb . [ 75%]
20:09:44  ....                                                                     [ 75%]
20:10:22  RavenPy-master/docs/notebooks/10_Data_assimilation.ipynb ........        [ 77%]
20:10:42  RavenPy-master/docs/notebooks/11_Climatological_ESP_forecasting.ipynb .. [ 77%]
20:11:09  ......                                                                   [ 78%]
20:11:27  RavenPy-master/docs/notebooks/12_Performing_hindcasting_experiments.ipynb . [ 78%]
20:11:37  .......                                                                  [ 80%]
20:11:44  RavenPy-master/docs/notebooks/Assess_probabilistic_flood_risk.ipynb .... [ 80%]
20:12:12  ....                                                                     [ 81%]
20:12:34  RavenPy-master/docs/notebooks/Comparing_hindcasts_and_ESP_forecasts.ipynb . [ 81%]
20:12:53  .......                                                                  [ 83%]
20:12:57  RavenPy-master/docs/notebooks/Distributed_hydrological_modelling.ipynb . [ 83%]
20:13:18  ......                                                                   [ 84%]
20:13:40  RavenPy-master/docs/notebooks/Hydrological_realtime_forecasting.ipynb .. [ 84%]
20:13:49  ....                                                                     [ 85%]
20:14:01  RavenPy-master/docs/notebooks/Managing_Jupyter_Environments.ipynb ...    [ 86%]
20:14:30  RavenPy-master/docs/notebooks/Perform_Regionalization.ipynb .......      [ 87%]
20:14:30  RavenPy-master/docs/notebooks/Running_HMETS_with_CANOPEX_dataset.ipynb . [ 87%]
20:14:53  .............                                                            [ 89%]
20:15:13  RavenPy-master/docs/notebooks/Sensitivity_analysis.ipynb .....           [ 90%]
20:15:21  RavenPy-master/docs/notebooks/time_series_analysis.ipynb ...........     [ 92%]
20:15:43  RavenPy-master/docs/notebooks/paper/Perform_a_climate_change_impact_study_on_a_watershed.ipynb . [ 93%]
20:22:42  ....................                                                     [ 96%]
20:22:44  notebooks/hummingbird.ipynb ............                                 [ 98%]
20:25:01  notebooks/stress-tests.ipynb ......                                      [100%]
20:25:01  
20:25:01  =============================== warnings summary ===============================
20:25:01  PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb:Cell 4:0
20:25:01    /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master/PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb:Cell 4:0: UserWarning: Conflicting comment markers found, using the latest:  NBVAL_IGNORE_OUTPUT VS NBVAL_IGNORE_OUTPUT
20:25:01  
20:25:01  -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
20:25:01  ============ 544 passed, 2 skipped, 1 warning in 4599.87s (1:16:39) ============

[tlvu]

Yesterday (2025-02-26), we got our first nightly end-2-end run that passed entirely in a very long time. 🎉

Thanks for the confirmation on your side. That's why I closed bird-house/birdhouse-deploy#496 yesterday because I expected all the noteoboks should now be passing.

To be noted that I did NOT apply any change mentioned in #142 (comment). It just ran with the usual references. Therefore, I have no idea why the notebooks listed in https://raw.githubusercontent.com/Ouranosinc/PAVICS-e2e-workflow-tests/refs/heads/master/test-override/jenkins-params-default.include.sh ran successfully this time. 🤔

Like I already mentionned in comment #142 (comment) above:

"For us, we will also leverage this mechanism to disable some more notebooks in the "generate output using jupyter nbconvert" phase. Your pipeline do not enable that phase so you do not need it."