feat(pip_requirements): support hashes

[peterbe]

What Renovate type are you using?
Renovate GitHub App (I think)

Describe the bug
See mozilla-services/tecken#1040

I have a project with renovate and by default Python support should be enabled but PRs on the requirements.txt is not being created.
The requirements.txt definitely has a format that both pip and hashin supports. Nothing weird there.

To Reproduce
See above. :)

Expected behavior
More PRs for outdated Python packages in the the requirements.txt

Additional context
Again; mozilla-services/tecken#1040 (comment)

[rarkins]

Can you describe what your expectations are for hash updating? Eg from which sources (public pypi only?) and updated/checked only when the version is changed?

[peterbe]

For example, psycopg2==2.7.4 isn't the latest hotness these days. According to https://pypi.org/pypi/psycopg2/json (see .info.version) all the rage these days is version 2.7.5. Also, if you look at .releases['2.7.5'] in that JSON you can get a complete list of all every release and each .digests.sha256 has the hashes needed.

If you, locally, run hashin on the requirements.txt you get https://gist.github.com/peterbe/39c294088cd3fc505dd4bf924dce7962

[peterbe]

I guess, ultimately, is that I'd like to see a PR like the one that hashin makes.

[rarkins]

If the hashes are available from pypi results then it should be easy. hashin describes a complicated process, eg downloading and extracting files - is that the “old” way before pypi stored the hashes? Otherwise if we need to use hashin then it’s a bit more work.

[rarkins]

BTW I will rename this issue because python support is enabled - it’s just that it doesn’t support hashes in requirements.txt. I will use this issue instead to track support for hashin-style hash updates. We can reuse existing terminology of “digests” for this

[peterbe]

hashin describes a complicated process, eg downloading and extracting files

It used to do that. Not any more. Not as of version 0.13 when it started using pypi.org instead. See https://github.com/peterbe/hashin/pull/52/files#diff-ee0c3f1d4738b7c444698b25746e26eaR349

[rarkins]

So if we base Renovate’s solution 100% off PyPi json then that would satisfy your requirements?

[rarkins]

How/where is the algorithm specified? Or can we just use the default?

[peterbe]

However, one thing I need to figure out is this...

If I personally run hashin here on my laptop I know exactly what version of Python I'm using. So to not get all the hashes for python2.7, python3.3 etc. I can type hashin -p cp36 psycopg2 (for CPython 3.6)

▶ hashin psycopg2==2.7.4  -r /tmp/reqs.txt -p cp36 -v
https://pypi.org/pypi/psycopg2/json
* Found hash for https://files.pythonhosted.org/packages/8c/a5/0e61d6f4a140a6e06a9ba40266c4b49123d834f1f97fe9a5ae0b6e45112b/psycopg2-2.7.4-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
*   Hash b68e89bb086a9476fa85298caab43f92d0a6af135a5f433d1f6b6d82cafa7b55
* Found hash for https://files.pythonhosted.org/packages/25/7e/eb6d6b1b5e99440def358f45f852f5ac62d26c59fef043770c1d7404c402/psycopg2-2.7.4-cp36-cp36m-manylinux1_i686.whl
*   Hash 0b9851e798bae024ed1a2a6377a8dab4b8a128a56ed406f572f9f06194e4b275
* Found hash for https://files.pythonhosted.org/packages/92/15/92b5c363243376ce9cb879bbec561bba196694eb663a6937b4cb967e230e/psycopg2-2.7.4-cp36-cp36m-manylinux1_x86_64.whl
*   Hash 733166464598c239323142c071fa4c9b91c14359176e5ae7e202db6bcc1d2eb5
* Found hash for https://files.pythonhosted.org/packages/00/95/4c5d19affca312e1c06d4f88241ebc564bf5269addd191ec4962f0c93553/psycopg2-2.7.4-cp36-cp36m-win32.whl
*   Hash ad75fe10bea19ad2188c5cb5fc4cdf53ee808d9b44578c94a3cd1e9fc2beb656
* Found hash for https://files.pythonhosted.org/packages/f9/77/e29b792740ddec37a2d49431efa6c707cf3869c0cc7f28c7411bb6e96d91/psycopg2-2.7.4-cp36-cp36m-win_amd64.whl
*   Hash 8966829cb0d21a08a3c5ac971a2eb67c3927ae27c247300a8476554cc0ce2ae8
* Editing /tmp/reqs.txt

However, if you don't supply the -p cp36 and Renovate (or some bot like Renovate) all they get is a requirements.txt file which they parse and find psycopg2==2.7.4 and decide "That can be upgraded!"
How does it know what Python version to run that for?

If the original maker of that requirements.txt manually used hashin (on her computer) with the -p cp36 how are we to know? If you just use hashin psycopg2==2.7.5 this happens:

▶ cp /tmp/reqs.txt /tmp/reqs.txt.before
▶ hashin psycopg2==2.7.5  -r /tmp/reqs.txt  # Imagine this being Renovate
▶ diff /tmp/reqs.txt /tmp/reqs.txt.before
1,31c1,6
< psycopg2==2.7.5 \
<     --hash=sha256:0b9e48a1c1505699a64ac58815ca99104aacace8321e455072cee4f7fe7b2698 \
<     --hash=sha256:0f4c784e1b5a320efb434c66a50b8dd7e30a7dc047e8f45c0a8d2694bfe72781 \
<     --hash=sha256:0fdbaa32c9eb09ef09d425dc154628fca6fa69d2f7c1a33f889abb7e0efb3909 \
<     --hash=sha256:11fbf688d5c953c0a5ba625cc42dea9aeb2321942c7c5ed9341a68f865dc8cb1 \
<     --hash=sha256:19eaac4eb25ab078bd0f28304a0cb08702d120caadfe76bb1e6846ed1f68635e \
<     --hash=sha256:3232ec1a3bf4dba97fbf9b03ce12e4b6c1d01ea3c85773903a67ced725728232 \
<     --hash=sha256:36f8f9c216fcca048006f6dd60e4d3e6f406afde26cfb99e063f137070139eaf \
<     --hash=sha256:59c1a0e4f9abe970062ed35d0720935197800a7ef7a62b3a9e3a70588d9ca40b \
<     --hash=sha256:6506c5ff88750948c28d41852c09c5d2a49f51f28c6d90cbf1b6808e18c64e88 \
<     --hash=sha256:6bc3e68ee16f571681b8c0b6d5c0a77bef3c589012352b3f0cf5520e674e9d01 \
<     --hash=sha256:6dbbd7aabbc861eec6b910522534894d9dbb507d5819bc982032c3ea2e974f51 \
<     --hash=sha256:6e737915de826650d1a5f7ff4ac6cf888a26f021a647390ca7bafdba0e85462b \
<     --hash=sha256:6ed9b2cfe85abc720e8943c1808eeffd41daa73e18b7c1e1a228b0b91f768ccc \
<     --hash=sha256:711ec617ba453fdfc66616db2520db3a6d9a891e3bf62ef9aba4c95bb4e61230 \
<     --hash=sha256:844dacdf7530c5c612718cf12bc001f59b2d9329d35b495f1ff25045161aa6af \
<     --hash=sha256:86b52e146da13c896e50c5a3341a9448151f1092b1a4153e425d1e8b62fec508 \
<     --hash=sha256:985c06c2a0f227131733ae58d6a541a5bc8b665e7305494782bebdb74202b793 \
<     --hash=sha256:a86dfe45f4f9c55b1a2312ff20a59b30da8d39c0e8821d00018372a2a177098f \
<     --hash=sha256:aa3cd07f7f7e3183b63d48300666f920828a9dbd7d7ec53d450df2c4953687a9 \
<     --hash=sha256:b1964ed645ef8317806d615d9ff006c0dadc09dfc54b99ae67f9ba7a1ec9d5d2 \
<     --hash=sha256:b2abbff9e4141484bb89b96eb8eae186d77bc6d5ffbec6b01783ee5c3c467351 \
<     --hash=sha256:cc33c3a90492e21713260095f02b12bee02b8d1f2c03a221d763ce04fa90e2e9 \
<     --hash=sha256:d7de3bf0986d777807611c36e809b77a13bf1888f5c8db0ebf24b47a52d10726 \
<     --hash=sha256:db5e3c52576cc5b93a959a03ccc3b02cb8f0af1fbbdc80645f7a215f0b864f3a \
<     --hash=sha256:e168aa795ffbb11379c942cf95bf813c7db9aa55538eb61de8c6815e092416f5 \
<     --hash=sha256:e9ca911f8e2d3117e5241d5fa9aaa991cb22fb0792627eeada47425d706b5ec8 \
<     --hash=sha256:eccf962d41ca46e6326b97c8fe0a6687b58dfc1a5f6540ed071ff1474cea749e \
<     --hash=sha256:efa19deae6b9e504a74347fe5e25c2cb9343766c489c2ae921b05f37338b18d1 \
<     --hash=sha256:f4b0460a21f784abe17b496f66e74157a6c36116fa86da8bf6aa028b9e8ad5fe \
<     --hash=sha256:f93d508ca64d924d478fb11e272e09524698f0c581d9032e68958cfbdd41faef
---
> psycopg2==2.7.4 \
>     --hash=sha256:0b9851e798bae024ed1a2a6377a8dab4b8a128a56ed406f572f9f06194e4b275 \
>     --hash=sha256:733166464598c239323142c071fa4c9b91c14359176e5ae7e202db6bcc1d2eb5 \
>     --hash=sha256:8966829cb0d21a08a3c5ac971a2eb67c3927ae27c247300a8476554cc0ce2ae8 \
>     --hash=sha256:ad75fe10bea19ad2188c5cb5fc4cdf53ee808d9b44578c94a3cd1e9fc2beb656 \
>     --hash=sha256:b68e89bb086a9476fa85298caab43f92d0a6af135a5f433d1f6b6d82cafa7b55

Now all the "junk" hashes are back for this version upgrade :(

[rarkins]

Sounds like we need python version awareness first and then solve hashes second. Is there a standardised way to specify python version in a repository or is it something we’d need users to add to their Renovate config? Right now we just take the latest available on pypi

[peterbe]

So if we base Renovate’s solution 100% off PyPi json then that would satisfy your requirements?

Yes. I don't know what Pyup uses internally but they can cope with requirements.txt files like this. I just prefer to use Renovate

How/where is the algorithm specified? Or can we just use the default?

What hashin does is that it defaults to sha256 unless it's changed by the -a/--algorithm CLI flag.
That's kinda silly because the user might prefer sha384 and that'd change edits from sha384 to sha256 if the user didn't remember to specify -a sha384.

[peterbe]

Sounds like we need python version awareness first and then solve hashes second.

How so? How does the current Renovate Python support work then?

Is there a standardised way to specify python version in a repository

If the project doesn't have a requirements.pip file, the requirements.txt file is sane and should contain the exact version after the ==.

If a requirements.txt file doesn't have hashes you can see lines like this:

Django>=2.1
psycopg2>2.7.0,<3.0

In that case, it's the same as good old package.json where you parse the operator and suggest upgrades accordingly.

[rarkins]

We have Python support, we just don’t support varying the results based on what version of Python the user wants, eg 3.6

[peterbe]

Ah. hashin doesn't either in a sense. That's why I filed: peterbe/hashin#75

That way upgrades would be done using the same specified Python versions (and/or algorithm of choice) you used when you added the package to the requirements.txt initially.

[zharinov]

@rarkins I think it's not necessary to know exact Python version, we can infer it from dep's currentDigest.

UPD: it seems the python version won't help since it also depends on operating system and so on

[zharinov]

I have clear model on how to handle this, just need some guidance on better implementation: (1) create pip-specific field for hashes that will travel from extract to update, (2) extend current digest handling to work with collections instead of single value, or (3) extract deps for each hash and group them somehow.

Ideally I would prefer (3) and will try it first.

[caphrim007]

@rarkins can I ask (purely out of curiosity) why @zharinov 's PR was closed? Is there something that outside interested parties could add that would satisfy that PR and this issue? Thank you both for the work that has gone into this.

[rarkins]

We changed the architecture for how we do updates. I think now what we should do is:

1. Ignore the hashes when we extract (focus on versions/constraints only)
2. Ignore the hashes when we replace (use auto-replace)
3. Add getArtifacts() to the pip_requirements manager to update hashes after

The main question is whether we update the hashes using our own code or using a third party tool like hashin

[rarkins]

@peterbe can you give us an update on your current requirements? Some of the links above now 404

[caphrim007]

@rarkins I'd like to add to the discussion since I have an interest in this feature.

I've created a sample repo using an actual set of requirements as they are laid out in a project I work on. It can be found here

https://github.com/caphrim007/sample-requirements

I think my use case (may?) be more simple than @peterbe ?

I don't have a need to specify the python version, the architecture, or the distribution method (source, wheel, etc). My workflow today which I had intended on having renovate do instead is simply

1. pip list --outdated --format --json
2. get package name (ex. foo==1.2.3)
3. find package in file
4. run hashin foo==1.2.3 -r file-that-was-found
5. create PR with updated files

If I can offer any other assistance or feedback, lemme know. Thanks!

[rarkins]

I think this should work. I checked out your example repository, manually updated tox to 3.15.2 in bootstrap.txt, then ran hashin tox==3.15.2 -r bootstrap.txt and it worked fine:

[caphrim007]

that is what I would expect to see, yes. Based off of the testing I did last night (and what @zharinov had in their PR) at present, renovate would not be able to detect the package name/hashes/ etc.

But what you've posted in the pic is essentially what I wanted to get out of renovate.

I figured this was the simplest case before tossing in features like python versions, archs, etc. Those too may be desirable at some point in the future. However, today, the basic case is sufficient for my own needs.

Not sure about @peterbe's reqs

[rarkins]

@caphrim007 do you know why if you run hashin readme_renderer==26.0 -r constraints.txt that it appends it to the file instead of updates the existing version?

[rarkins]

Other than that, it's essentially working: https://github.com/renovate-tests/sample-requirements/pulls

[caphrim007]

If I had to guess it's due to some quirk in how naming happens with some dependencies in pypi and req files.

I think I have a hack in one of my own scripts to handle this case actually. Let me just verify that though.

In a nutshell, it tries to find foo_package, but the dependency is actually foo-package but for whatever reason, the package online is named by hyphen or dash. I can't really explain it because I'm not entirely sure what is happening, but that's my guess.

Lemme take a look at my script/hack workaround. I don't think this is a common thing though

[caphrim007]

@rarkins what you're seeing appears to be what I was hacking around in my local scripts. pypi...somewhere....draws some sort of distinction between underscored and hyphenated packages.

Here's a specific package I was hacking around; https://pypi.org/project/websocket_client/

I'm not sure what the rules are for when to convert between underscored and hyphenated, orther than than underscores are "discouraged" for packages and pypi may be forcing a conversion on your behalf if you submit an underscored package name.

https://www.python.org/dev/peps/pep-0008/#package-and-module-names

That's my best guess.

When I update the package you mentioned, it gets appended to my constraint file as readme-renderer. Likely because it was looking for the underscored name and didn't find it.

On the bright side, for this package, the hashes for the hyphenated version are the same as for the underscored version.

So, a couple of proposals.

First, I can reach out to the packaging working group/PyPA folks and ask for some clarification around what is expected (hyphen vs underscore) and why both work when you install them. I imagine its for reasons of not breaking the world though. They likely made a breaking change and didn't want to affect all of the existing software that likely hardcode the underscored name.

Second, renovate can make some judgement calls on how it wants to handle these. If PEP8 is the reason that pypi did this, then PEP8 would be considered correct form and it may be the case that renovate would want to do this sort of conversion for the user automatically and replace the underscored name in the requirements file.

In case that behavior is making too many assumptions though, maybe this is a configurable option in the pip_requirements manager.

Whether or not conversion is done, it would be useful for debugging purposes to see when renovate detects the oddity. For instance, a logger.debug message when it sees the package name that includes an underscore.

Perhaps it should ignore packages with underscores, or include a different message in the PR. I'll leave that for you to decide.

In this case with my own dependencies I verified that the readme_renderer, when changed to hyphenated, does not result in a failed pip install. So I consider this a bug in my requirements and I'll change it. I did the same verification for the websocket-client dependency and changed that too.

Lemme know if you care to have me ask the PyPA folks about the underscore/hyphen thing. Otherwise, I'll have a look at the PR you posted here and see if I get the same results.

[rarkins]

I think is beyond the scope of what Renovate should be responsible for. e.g. if hashin has a fault, or users should prefer hyphen over underscore but don't, then there's not much Renovate can do until it's fixed in hashin or pip or wherever necessary.

[peterbe]

@caphrim007 I don't mean to sound lazy but can you, for a moment, look into whether this can be fixed in hashin (hi! I'm its maintainer) so that hashin becomes conveniently agnostic?
I think it's perfectly reasonable that a user might have...

python-memcached==1.59 \
    --hash=sha256:4dac64916871bd3550263323fc2ce18e1e439080a2d5670c594cf3118d99b594 \
    --hash=sha256:a2e28637be13ee0bf1a8b6843e7490f9456fd3f2a4cb60471733c7b5d5557e4f

in their requirements file but forgot that and typed:

hashin python_memcached -r requirements.txt

or, the other way around. That it was written as python_memcached already in requirements.txt but now you're asking to upgrade python-memcached.

I don't know how hard or easy it would be but it certainly seems like a nice API that hashin copes so that you don't end up with this after you've run hashin -u or hashin -u -i:

python-memcached==1.59 \
    --hash=sha256:4dac64916871bd3550263323fc2ce18e1e439080a2d5670c594cf3118d99b594 \
    --hash=sha256:a2e28637be13ee0bf1a8b6843e7490f9456fd3f2a4cb60471733c7b5d5557e4f
python_memcached==1.60 \
    --hash=sha256:x00c64916871bd3550263323fc2ce18e1e439080a2d5670c594cf3118d99b594 \
    --hash=sha256:x02e28637be13ee0bf1a8b6843e7490f9456fd3f2a4cb60471733c7b5d5557e4f

Shall we? https://github.com/peterbe/hashin

I don't mind reviewing a PR and discussing but I must admit that I'd struggle to find time to take the lead on it at the moment.

[rarkins]

@peterbe it’s a little different. The user has x_y in their requirements file and we run hashin x_y==1.0.1 but hashin appends x-y==1.0.1 to the file and leaves the existing x_y in place.

[peterbe]

@peterbe it’s a little different. The user has x_y in their requirements file and we run hashin x_y==1.0.1 but hashin appends x-y==1.0.1 to the file and leaves the existing x_y in place.

My point is that if run hashin x_y==1.0.1 or hashin x-y==1.0.1 is should work the same. If the file already called it x_y it should continue that. If it called it x-y is should continue that too.

I.e. hashin doesn't try to correct anything.

[caphrim007]

@peterbe I can look into that with hashin. if I open an issue over there, do you have time to talk at me on how we want to implement it?

From renovate's perspective i think it can continue to just use hashin. @rarkins hasn't added any special handling to the renovate code, just at the moment it behaves in such a way that it would append the hyphenated/underscored version if it didn't find the other.

so i dont think renovate is really blocked here for anything. its just that we could enhance the behavior of hashin. thoughts?

[peterbe]

if I open an issue over there, do you have time to talk at me on how we want to implement it?

Yes indeed! I think the issue should be fairly straightforward. We basically want hashin to not over-correct but respect whatever the user's preferred "convention" is. Even if the user is "wrong" if you know what I mean.

[caphrim007]

@peterbe going to open that issue right now. For the renovate specific usage of this though, should we block this feature until hashin adds the enhancement? Or would this just be a dependency that is updated/installed automatically when renovate runs.

I noticed in @rarkins pr that it does a "pip install hashin".

[caphrim007]

@rarkins while I wait for the PR I have open for review in hashin, I was curious about @peterbe's other remarks about Python version.

For example, I have the following in a repo

{
  "pip_requirements": {
    "enabled": true,
    "fileMatch": [
      "requirements/.*\\.txt"
    ]
  },
  "enabledManagers": ["pip_requirements"]
}

I'm curious if there are any conventions that have been made in other managers to support manager specific options to trickle down to the manager, or if there is only a common set of options for all managers (enabled, fileMatch, etc).

[caphrim007]

Mousing around the code, I think what I'm wondering about is specifying Artifact configuration.

It does appear that some managers allow for that (I looked at bundler and composer), and that such options could provide input to hashin's -p argument.

However, these settings appear to apply to renovate's configuration, and not the configuration found in a source repo, is that correct? As such, one could configure renovate to apply "-p 3.7 -p 3.8" to all source repos with a pip_requirements manager specified, but the repo owner would be unable to control that configuration.

Am I reading that correctly?

[rarkins]

We do have a precedent of attempting compatibility of language versions including python. So far this is for package managers that have a field for specifying the python version though.

We have some precedent for package manager configuration options such as npmDedupe or one composer one, but we try to minimize those or else our list of config options will really blow up.

[caphrim007]

@rarkins,

@peterbe and I have resolved the issue you were seeing in the appending of package names. It is available in hashin 0.15.0.

At this point I would say it is acceptable to move forward on your draft implementation of hashes which you have linked in this issue. Let us know if there is anything else you need from us.

[renovate-release]

🎉 This issue has been resolved in version 21.17.0 🎉

The release is available on:

• GitHub release
• 21.17.0

Your semantic-release bot 📦🚀