FlexMeasures should not pin its dependencies in the Pypi distribution

[nhoening]

Problem statement

When one pip-installs FlexMeasures, the versions of dependencies are pinned exactly. In some cases, when FlexMeasures is part of an environment, this leads to version conflicts. A good example might be plugins, where people have their own dependencies, and these dependencies have dependencies. Pip will run into dependency conflicts. I document one in this issue.

One can see FlexMeasures not as the sole platform to be run, but as a dependency itself in some project.

Technical details

We provide ranges for version requirements in requirements/*.in. Based on these, we regularly (e.g. after each release) compile exact pins, which then live in requirements/*.txt.
So each FlexMeasures checkout as well as release has a working set of exact dependency versions, which are tested.
For the distribution on Pypi, setup.py makes the choice to read in requirements/*.txt in order to fill install_requires.

Proposed resolution

• install_requires should use requirements/*.in. Pro: FlexMeasures works better as dependency. Contra: People might experience errors which are harder to debug, e.g. when a new version of some dependency leads to a bug.
• The docker image would still be pinned exactly (see Dockerfile), as hosting FlexMeasures is the sole purpose of the containers based on it. Contra: I'm actually not sure if this leads to problems when plugin authors create Docker images based on this.

[victorgarcia98]

Nice! Certainly, dependencies can be a headache.

install_requires should use requirements/*.in. Pro: FlexMeasures works better as dependency. Contra: People might experience errors which are harder to debug, e.g. when a new version of some dependency leads to a bug.

To cope with hard to debug environments (e.g: closed source software that uses FlexMeasures), I think we can propose to include which modules are being instaled with flexmeasures in the GH template. We can other stuff such as OS, PostgreSQL version, ...

In order to make it easier for the users, we can propose to use, for instance, pipdeptree. pipdeptree is a pip installable utility that generates dependency trees. The interesting feature is that we can pass a python module as an input of the CLI and it will print the required and installed versions.

Moreover, we can use the flag -f to generate a pip compatible requirements.txt for that setup.

Example for the python-s2-protocol library:

Input:

$  pipdeptree -f -p python-s2-protocol

Output:

python-s2-protocol @ git+https://github.com/SeitaBV/python-s2-protocol@ef50ee8f5d9ce413f74ec66fd28ace690fb480a4
  jsonschema==4.15.0
    attrs==23.1.0
    pyrsistent==0.19.3
  pydantic==1.10.0
    typing_extensions==4.5.0
  pytest==7.2.2
    attrs==23.1.0
    exceptiongroup==1.1.1
    iniconfig==2.0.0
    packaging==23.1
    pluggy==1.0.0
    tomli==2.0.1
  requests==2.28.1
    certifi==2022.12.7
    charset-normalizer==2.1.1
    idna==3.4
    urllib3==1.26.15

[nhoening]

Thanks @victorgarcia98 - documenting debugging best practices would be great.

So in principle you are supporting this change idea?

[victorgarcia98]

Yes, I support it. I upvote this discussion.

[nhoening]

Done.