Check code style: Using pre-commit hooks (recommended) #933

NimaSarajpoor · 2022-05-06T17:44:58Z

NimaSarajpoor
May 6, 2022

Hello,

I have been trying to use pre-commit hooks for testing files with black and flake8, as recommended in the contribution part. But, I got into some problems that I cannot resolve. I will provide the process below:

1- I follow the steps but I got this error: error RuntimeError: failed to find interpreter for Builtin discover of python_spec='python3.9'
2- I noticed I am using python 3.8; however, [mlxtend](https://github.com/rasbt/mlxtend)/.pre-commit-config.yaml shows that it requires python 3.9
3- I couldn't upgrade my python through conda in spite of the fact that such package is available in anaconda. So, after some search, I realized if I create a virtual environment, I can install new version of python there...so:

4- conda create -n py310 python=3.10

5- I checked the python version and it was 3.10. However, again, I got error while doing the pre-commit steps. Apparently, I had to install 3.9 only.

6-I created another virtual environment with python 3.9... and then follow the steps provided in pre-commit steps...

7-It seems everything went well.

8-I did a simple commit to see how things would go: git commit --allow-empty -m "Empty test commit"
And, I got:

black................................................(no files to check)Skipped
flake8...............................................(no files to check)Skipped
isort (python).......................................(no files to check)Skipped
[master f9b6cf50] Empty test commit

So far, it seems things are good. Correct? Since I did --allow-empty (i.e. no actual changes in files), there were no files to be checked.

Ok, now let us try it on the some directories:

$ black mlxtend/frequent_patterns/apriori.py
Traceback (most recent call last):
  File "C:\Users\nimas\anaconda3\Scripts\black-script.py", line 5, in <module>
    from black import main
  File "src\black\__init__.py", line 41, in <module>
  File "src\black\nodes.py", line 36, in <module>
  File "src\blib2to3\pygram.py", line 186, in initialize
  File "src\blib2to3\pygram.py", line 32, in __init__
AttributeError: '_python_symbols' object has no attribute 'with_item'

Isn't it weird?

I tried black on another package, and it worked fine...

(stumpy) nimas (Traversal) stumpy
$ black stumpy/stump.py
All done! \u2728 \U0001f370 \u2728
1 file left unchanged.

I was wondering if someone can explain what is going on.

The same issue exists for flake8...

(stumpy) nimas (Traversal) stumpy
$ flake8 stumpy/stump.py
(stumpy) nimas (Traversal) stumpy
$

# so, it seems  fine here.

Now, in mlxtend....

$ flake8 mlxtend/frequent_patterns/apriori.py
Traceback (most recent call last):
  File "C:\Users\nimas\anaconda3\lib\site-packages\flake8\plugins\manager.py", line 157, in load_plugin
    self._load()
  File "C:\Users\nimas\anaconda3\lib\site-packages\flake8\plugins\manager.py", line 134, in _load
    self._plugin = self.entry_point.load()
  File "C:\Users\nimas\anaconda3\lib\importlib\metadata.py", line 77, in load
    module = import_module(match.group('module'))
  File "C:\Users\nimas\anaconda3\lib\importlib\__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "C:\Users\nimas\anaconda3\lib\site-packages\flake8_black.py", line 10, in <module>
    import black
  File "src\black\__init__.py", line 41, in <module>
  File "src\black\nodes.py", line 36, in <module>
  File "src\blib2to3\pygram.py", line 186, in initialize
  File "src\blib2to3\pygram.py", line 32, in __init__
AttributeError: '_python_symbols' object has no attribute 'with_item'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\nimas\anaconda3\Scripts\flake8-script.py", line 9, in <module>
    sys.exit(main())
  File "C:\Users\nimas\anaconda3\lib\site-packages\flake8\main\cli.py", line 22, in main
    app.run(argv)
  File "C:\Users\nimas\anaconda3\lib\site-packages\flake8\main\application.py", line 363, in run
    self._run(argv)
  File "C:\Users\nimas\anaconda3\lib\site-packages\flake8\main\application.py", line 350, in _run
    self.initialize(argv)
  File "C:\Users\nimas\anaconda3\lib\site-packages\flake8\main\application.py", line 330, in initialize
    self.find_plugins(config_finder)
  File "C:\Users\nimas\anaconda3\lib\site-packages\flake8\main\application.py", line 159, in find_plugins
    self.check_plugins.load_plugins()
  File "C:\Users\nimas\anaconda3\lib\site-packages\flake8\plugins\manager.py", line 415, in load_plugins
    plugins = list(self.manager.map(load_plugin))
  File "C:\Users\nimas\anaconda3\lib\site-packages\flake8\plugins\manager.py", line 302, in map
    yield func(self.plugins[name], *args, **kwargs)
  File "C:\Users\nimas\anaconda3\lib\site-packages\flake8\plugins\manager.py", line 413, in load_plugin
    return plugin.load_plugin()
  File "C:\Users\nimas\anaconda3\lib\site-packages\flake8\plugins\manager.py", line 164, in load_plugin
    raise failed_to_load
flake8.exceptions.FailedToLoadPlugin: Flake8 failed to load plugin "BLK" due to '_python_symbols' object has no attribute 'with_item'.

One last information that might help:
For stumpy, when I was out of virtual environment, I got this error once when I ran black:

Traceback (most recent call last):
  File "C:\Users\nimas\anaconda3\Scripts\black-script.py", line 5, in <module>
    from black import main
  File "src\black\__init__.py", line 41, in <module>
  File "src\black\nodes.py", line 36, in <module>
  File "src\blib2to3\pygram.py", line 186, in initialize
  File "src\blib2to3\pygram.py", line 32, in __init__
AttributeError: '_python_symbols' object has no attribute 'with_item'

Then, after talking to the owner(i.e. main maintainer) of package, we thought something is probably wrong in my environment. So, I created a virtual environment with help of environment.yml provided in the package, using this command:

conda env update --name myenv --file environment.yml --prune

If I remember correctly, I did the same for mlxtend when I got a similar error after running black and I tried to install packages in my virtual environment with environment.yml. But, that did not help, and I kept getting the same error. That is why I tried pre-commit hooks as recommended in the contribution section.

I am trying to understand what is happing inside my computer. I think things can go wrong when I sometime use pip and sometimes conda install.

It would be great if someone can provide me with a source (like a link to a weblog or something) that explains these specific stuff...like setting PATH for package, or pip-vs-conda, and stuff. It is really confusing...

Answered by rasbt

May 9, 2022

Nice, glad to hear it's finally working!

View full answer

rasbt · 2022-05-06T18:17:46Z

rasbt
May 6, 2022
Maintainer

Oh this sounds tricky! I actually just set up the black formatting and pre-commit hook stuff this week, and it's all pretty new so there may be some kinks.

Regarding Python 3.9, that's a good point. I didn't know this could cause issues, and I am happy to downgrade it to say Python 3.6 or 3.7. I am curious about the following though.

Let's assume you

Change 3.9 to 3.6 in your local version of https://github.com/rasbt/mlxtend/blob/master/.pre-commit-config.yaml
And then setup the pre-commit hook in your standard environment with Python 3.8. Would it work (Python 3.6 being a minimum version) or would it complain that the version does not match exactly?

Maybe if this works and you can run it in your standard environment, maybe then solves also these other issues you are getting. I think it's worth a try.

1 reply

NimaSarajpoor May 6, 2022
Author

@rasbt
Thanks for the suggestion. Unfortunately, that did not help.

$ pre-commit install
pre-commit installed at .git\hooks\pre-commit
(mlxtend_env) nimas (warning) mlxtend
$ git commit --allow-empty -m "Empty test commit"
[INFO] Installing environment for https://github.com/psf/black.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
An unexpected error has occurred: CalledProcessError: command: ('C:\\Users\\nimas\\anaconda3\\python.exe', '-mvirtualenv', 'C:\\Users\\nimas\\.cache\\pre-commit\\repoe7l54aso\\py_env-python3.6', '-p', 'python3.6')
return code: 1
expected return code: 0
stdout:
    RuntimeError: failed to find interpreter for Builtin discover of python_spec='python3.6'

stderr: (none)
Check the log at C:\Users\nimas\.cache\pre-commit\pre-commit.log

And, my python is 3.8:

(mlxtend_env) nimas (warning) mlxtend
$ python -V
Python 3.8.5

And, pre-commit log file:

### version information

...
pre-commit version: 2.19.0
git --version: git version 2.23.0.windows.1
sys.version:
    3.8.5 (default, Sep  3 2020, 21:29:08) [MSC v.1916 64 bit (AMD64)]
sys.executable: C:\Users\nimas\anaconda3\python.exe
os.name: nt
sys.platform: win32
...

### error information

...
An unexpected error has occurred: CalledProcessError: command: ('C:\\Users\\nimas\\anaconda3\\python.exe', '-mvirtualenv', 'C:\\Users\\nimas\\.cache\\pre-commit\\repoe7l54aso\\py_env-python3.6', '-p', 'python3.6')
return code: 1
expected return code: 0
stdout:
    RuntimeError: failed to find interpreter for Builtin discover of python_spec='python3.6'
    
stderr: (none)
...

...
Traceback (most recent call last):
  File "C:\Users\nimas\anaconda3\lib\site-packages\pre_commit\error_handler.py", line 73, in error_handler
    yield
  File "C:\Users\nimas\anaconda3\lib\site-packages\pre_commit\main.py", line 361, in main
    return hook_impl(
  File "C:\Users\nimas\anaconda3\lib\site-packages\pre_commit\commands\hook_impl.py", line 238, in hook_impl
    return retv | run(config, store, ns)
  File "C:\Users\nimas\anaconda3\lib\site-packages\pre_commit\commands\run.py", line 414, in run
    install_hook_envs(to_install, store)
  File "C:\Users\nimas\anaconda3\lib\site-packages\pre_commit\repository.py", line 223, in install_hook_envs
    _hook_install(hook)
  File "C:\Users\nimas\anaconda3\lib\site-packages\pre_commit\repository.py", line 79, in _hook_install
    lang.install_environment(
  File "C:\Users\nimas\anaconda3\lib\site-packages\pre_commit\languages\python.py", line 219, in install_environment
    cmd_output_b(*venv_cmd, cwd='/')
  File "C:\Users\nimas\anaconda3\lib\site-packages\pre_commit\util.py", line 146, in cmd_output_b
    raise CalledProcessError(returncode, cmd, retcode, stdout_b, stderr_b)
pre_commit.util.CalledProcessError: command: ('C:\\Users\\nimas\\anaconda3\\python.exe', '-mvirtualenv', 'C:\\Users\\nimas\\.cache\\pre-commit\\repoe7l54aso\\py_env-python3.6', '-p', 'python3.6')
return code: 1
expected return code: 0
stdout:
    RuntimeError: failed to find interpreter for Builtin discover of python_spec='python3.6'
    
stderr: (none)
...

rasbt · 2022-05-06T19:07:21Z

rasbt
May 6, 2022
Maintainer

Oh I have an idea, how about replacing python3.9 with python3.x?

4 replies

NimaSarajpoor May 6, 2022
Author

Nope. Didn't help. I also deleted my local mlxtend, and git clone it again. This time, I did not even create a virtual environment, All things are in conda base now...

Then:

1-I created a branch "warning"....
2-modify the file .pre-commit-config.yaml: python3.9 --> python3.x
3-install pacakge mlxtend:

(base) nimas (warning) mlxtend
$ python setup.py install

4-Then after following the steps of pre-commit .... I got similar error....

# after following steps of pre-commit provided in the contribution section

(base) nimas (warning) mlxtend
$ git commit --allow-empty -m "Empty test commit"
[INFO] Installing environment for https://github.com/psf/black.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
An unexpected error has occurred: CalledProcessError: command: ('C:\\Users\\nimas\\anaconda3\\python.exe', '-mvirtualenv', 'C:\\Users\\nimas\\.cache\\pre-commit\\repoe7l54aso\\py_env-python3.x', '-p', 'python3.x')
return code: 1
expected return code: 0
stdout:
    RuntimeError: failed to find interpreter for Builtin discover of python_spec='python3.x'

stderr: (none)
Check the log at C:\Users\nimas\.cache\pre-commit\pre-commit.log

NimaSarajpoor May 6, 2022
Author

Maybe I should wipe out anaconda and re-install it again...or maybe I shouldn't use anaconda....to be honest, I have no idea what is going on :)

rasbt May 6, 2022
Maintainer

Arg, sorry for making this so complicated. But I wonder if it would work if you just remove the language_version

I.e.,

repos:
-   repo: https://github.com/psf/black
    rev: 22.3.0
    hooks:
    - id: black
      language_version: python3.9
-   repo: https://gitlab.com/pycqa/flake8
    rev: 3.9.2
    hooks:
    - id: flake8
- repo: https://github.com/pycqa/isort
  rev: 5.10.1
  hooks:
    - id: isort
      name: isort (python)
      args: ["--profile", "black"]

Becomes

repos:
-   repo: https://github.com/psf/black
    rev: 22.3.0
    hooks:
    - id: black
-   repo: https://gitlab.com/pycqa/flake8
    rev: 3.9.2
    hooks:
    - id: flake8
- repo: https://github.com/pycqa/isort
  rev: 5.10.1
  hooks:
    - id: isort
      name: isort (python)
      args: ["--profile", "black"]

NimaSarajpoor May 6, 2022
Author

I am actually learning!...so it is all good :)

It helps partially...I mean now, it runs black / flake8 for commit. But, I cannot directly run black on a .py file...

please see below:

$ git commit -m "Remove python version for Black"
[INFO] Installing environment for https://github.com/psf/black.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for https://gitlab.com/pycqa/flake8.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for https://github.com/pycqa/isort.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
black................................................(no files to check)Skipped
flake8...............................................(no files to check)Skipped
isort (python).......................................(no files to check)Skipped
[warning 24d410e2] Remove python version for Black
 1 file changed, 1 deletion(-)
(base) nimas (warning) mlxtend
$ black mlxtend/frequent_patterns/apriori.py
Traceback (most recent call last):
  File "C:\Users\nimas\anaconda3\Scripts\black-script.py", line 5, in <module>
    from black import main
  File "src\black\__init__.py", line 41, in <module>
  File "src\black\nodes.py", line 36, in <module>
  File "src\blib2to3\pygram.py", line 186, in initialize
  File "src\blib2to3\pygram.py", line 32, in __init__
AttributeError: '_python_symbols' object has no attribute 'with_item'

Maybe I shouldn't be worried about it. But, I am just curious what is going on here. Because, I think, in both cases, i.e. directly running black on .py file or checking black format while doing commit, it uses the same black version installed in my system.

Do you think I should submit an issue for this on "Black" repo?

rasbt · 2022-05-08T21:11:15Z

rasbt
May 8, 2022
Maintainer

Huh, that's weird. Never seen something like this before. Can you check if this is a current version of black? The src\blib2to3\pygram.py sounds like maybe there is something with Python 2.x involved. You can check via

black --version

1 reply

NimaSarajpoor May 8, 2022
Author

I ran black --version and got similar error! So, I started doing pip uninstall black repeatedly to make sure there is no black package in my system. I then did pip install black, and now everything is fine and working!!

Thanks a lot for your help!!!

rasbt · 2022-05-09T00:57:38Z

rasbt
May 9, 2022
Maintainer

Nice, glad to hear it's finally working!

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Check code style: Using pre-commit hooks (recommended) #933

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 4 comments 6 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

Select a reply

Check code style: Using pre-commit hooks (recommended) #933

NimaSarajpoor May 6, 2022

Replies: 4 comments · 6 replies

rasbt May 6, 2022 Maintainer

NimaSarajpoor May 6, 2022 Author

rasbt May 6, 2022 Maintainer

NimaSarajpoor May 6, 2022 Author

NimaSarajpoor May 6, 2022 Author

rasbt May 6, 2022 Maintainer

NimaSarajpoor May 6, 2022 Author

rasbt May 8, 2022 Maintainer

NimaSarajpoor May 8, 2022 Author

rasbt May 9, 2022 Maintainer

NimaSarajpoor
May 6, 2022

Replies: 4 comments 6 replies

rasbt
May 6, 2022
Maintainer

NimaSarajpoor May 6, 2022
Author

rasbt
May 6, 2022
Maintainer

NimaSarajpoor May 6, 2022
Author

NimaSarajpoor May 6, 2022
Author

rasbt May 6, 2022
Maintainer

NimaSarajpoor May 6, 2022
Author

rasbt
May 8, 2022
Maintainer

NimaSarajpoor May 8, 2022
Author

rasbt
May 9, 2022
Maintainer