config: add new DEFAULT_CONFIG_PARAMETERS_SCRIPT_URL

[tlvu]

Require Ouranosinc/PAVICS-e2e-workflow-tests#142

[fmigneault]

Waiting on Ouranosinc/PAVICS-e2e-workflow-tests#142 deliberation.

Will have to check with @perronld also about the impact.

If we need to reboot the compose each time a change is applied to the script to take effect, this will not be a viable solution. On the other hand, if it doesn't need a reboot (ie: it applies dynamically), why is there even a need to supply it here? It couldn't just be a variable in the existing scripts that toggle/filter based on a "blacklist" of notebooks?

[tlvu]

If we need to reboot the compose each time a change is applied to the script to take effect, this will not be a viable solution.

No need to reboot Jenkins each time. This is dynamic on each build request.

On the other hand, if it doesn't need a reboot (ie: it applies dynamically), why is there even a need to supply it here? It couldn't just be a variable in the existing scripts that toggle/filter based on a "blacklist" of notebooks?

Good question ! I did not implement this just for blacklisting notebooks. I implemented a generic pluggable default override that can do anything (blacklist, adding new, performing some pre/post processing, ... ).

The intend is a generic solution so we do not have add more commits each time we need yet a new feature and spawn again a new variable to toogle and that would work for any repos, even those unknown at this point by the current Jenkins config.

Each change is trying to approach us to resolving this issue Ouranosinc/PAVICS-e2e-workflow-tests#57

[fmigneault]

I'm somewhat worried about the feature in the long term.

For now, we already have a way to override some variables "in-line" to an execution with Jenkins parameters, and a way to define a different branch for PAVICS-e2e-workflow-tests. This should provide enough leverage to test alternative behaviors.

If we have situations where more code is needed to "undo" certain operations to handle edge cases, I'd argue that the problem lies in the main PAVICS-e2e-workflow-tests scripts in the first place. Hacking them via another default config, which will probably not suit our needs at some point, so it would need another test-override config, just means we'll have many places where code doing some test preparation is defined. Those "multi-layered patches" will never be merged back into the PAVICS-e2e-workflow-tests scripts, and everything will just be harder to maintain or understand between us, because nothing runs the same way anywhere.

[tlvu]

For now, we already have a way to override some variables "in-line" to an execution with Jenkins parameters, and a way to define a different branch for PAVICS-e2e-workflow-tests. This should provide enough leverage to test alternative behaviors.

I see the confusion ! You can do it because your pipeline have access to PAVICS' env.local so you override those Jenkins params programmatically in env.local.

But in practice, outside of your pipeline, there is no way for a regular user to perform this programmatic override because a standalone Jenkins do not have access to the target env.local. That feature is a way for regular user to perform programmatic override !

Those "multi-layered patches" will never be merged back into the PAVICS-e2e-workflow-tests scripts, and everything will just be harder to maintain or understand between us, because nothing runs the same way anywhere.

Well isn't this the same with our birdhouse-deploy? Any component is like a "multi-layered patches" because any subsequent component can override anything (variables, docker-compose directives) in all previous components. We also have a bunch of custom private components in external repos that we also will never merge back to birdhouse-deploy because it's specific to us !

Our PAVICS deployment is slightly different in term of components enabled but once a component is enabled, they have the same config, except for the Jupyter image list of course, we have a lot more. I think the "nothing runs the same way anywhere" is how we ensure each org can customize PAVICS to their liking. Same for this Jenkins I hope.

[fmigneault]

Well isn't this the same with our birdhouse-deploy? [...]

I do not question that aspect regarding the custom activation of certain components, their customization or even the addition of extra components by certain organizations. In that sense, having the script to configure the test pipeline that targets those custom components is totally fine.

The problem I foresee is more about the context under which this PR was opened, that is, disabling certain tests that feels "patchy". I find if custom modifications to applicable tests of (supposedly) base components must be ignored, they are fundamentally broken components/tests. Having too many custom things in place will just make it such that their reproducibility is questionable at best, and proper fixes will remain in custom scripts rather than applied to the main repository (aka. the discrepancies we highlight must be avoided in https://birdhouse-deploy.readthedocs.io/en/latest/contributing.html#birdhouse-multi-organization-git-repository-management).

However, I hear the concern for other "legit" situations, especially given that you might not have the pipeline overrides via env.local on your end. Given the default is empty and only the Ouranos-specifc jcasc is edited, the feature is fine on its own. I just hope that script is not "abused" by other cases as the ignored-tests mentioned above.

[tlvu]

Given the default is empty and only the Ouranos-specifc jcasc is edited, the feature is fine on its own.

Exact, the default is empty and only the Ouranos-specifc jcasc is edited so we will be responsible for maintaining our own "exception" list.

Only us will need to use it. For other, this serve simply as example/sample code. But since we use it for real, we can make sure the sample code actually works.

I just hope that script is not "abused" by other cases as the ignored-tests mentioned above.

Again no "abuse" possible since someone has to activate this explicitly, manually. Nothing will "creep in" unnoticed. All changes are sourced controlled anyways, unless someone write their own override and not use our override.