How to use package_dir and correctly include docs and extra files in a distribution?

[cdfarrow]

I am updating my project to use the latest setuptools configuration using setup.cfg and pyproject.toml. I want to use a slightly different directory structure (a variant on the "src" style including user docs and other necessary files), and I'm sure it should be possible, but I'm struggling to understand how the package_dir parameter works together with packages, include_package_data, etc.

I've found this section on docs.python.org, which is the most detailed explanation that I've found. This page explains package_dir in a different way. However, I'm still left scratching my head trying to understand how everything works. I think it is a case where everyone 'in the know' understands the implicit workings, but newcomers like me are not able to connect the dots in the documentation.

In any case, I've finally figured out that in examples like

package_dir =
    mypkg = mypkg

the two strings 'mypkg' are actually referring to very different things. The first one is the directory name that will be created under site-packages (and hence imported with 'import mypkg'), while the second one is referring to the source directory for the Python package, and thus the source of the files that will actually be copied into that mypkg installation.

So in my config below I have

package_dir =
    flexlibs = flexlibs

to generate a flexlibs package in site-packages, which includes the code\ and tests\ directories and the top-level __init__.py for the Python package.

In addition, I have the docs\ and dlls\ directories specified in MANIFEST.in, and include_package_data = True. This now generates the right directory structure in the wheel (exactly matching the flexlibs sub-tree below). However, it complains that it can't find __init__.py files in the docs\ directory tree. So, it seems I still don't have things right.

Can someone please explain how to configure this correctly to include the docs\ and dlls\ directories as package data with no warnings, and also explain how it works and how the different configuration parameters work together.

Source tree:

├───MANIFEST.in
├───pyproject.toml
├───setup.cfg
├───docs
│   └───sphinx
└───flexlibs
    ├───__init__.py
    ├───code
    │   ├───__init__.py
    │   ├───FLExInit.py
    │   └───FLExProject.py
    ├───docs
    │   └───flexlibsAPI
    ├───dlls
    │   ├───A.dll
    │   ├───B.dll
    │   └───readme.txt
    └───tests
        ├───test_FLExInit.py
        └───test_FLExProject.py

The Sphinx docs are compiled into flexlibs\docs\flexlibsAPI as user documentation.

setup.cfg snippet:

package_dir = 
    flexlibs=flexlibs
    
# The dlls & docs directories are specified in MANIFEST.in
# 'include_package_data = True' includes these in the wheel.
include_package_data = True

MANIFEST.in:

graft flexlibs\docs
graft flexlibs\dlls

Thanks for your help!

[abravalheri]

Hi @cdfarrow, can you confirm what is the expected directory structure that you want to have installed in your site-packages directory after you pip install your package?

Is it the case that you want to have the following?

site-packages
└───flexlibs
    ├───__init__.py
    ├───code
    │   ├───__init__.py
    │   ├───FLExInit.py
    │   └───FLExProject.py
    ├───docs
    │   └───flexlibsAPI
    ├───dlls
    │   ├───A.dll
    │   ├───B.dll
    │   └───readme.txt
    └───tests
        ├───test_FLExInit.py
        └───test_FLExProject.py

(This means that you flexlibs/__init__.py file performs the relevant imports otherwise your users will have to do from flexlibs.code import ...)

If that is the case, you don't need to specify package-dir.

If you also don't include packages= or py_modules= configuration, setuptools will automatically auto-discovery the relevant directories and packages. Please note however that if you have non-conventional folders at the root of your project (siblings to the flexlibs package folder) you might need to personalise package discovery like the following:

# setup.cfg
...

[options]
...
include_package_data = True
packages = find_namespace:
# Use find_namespace: instead of find: to opt into PEP 420 behaviour

[options.packages.find]
include = flexlibs*

...

[cdfarrow]

Thanks. Yes, that is the structure I want in site-packages; and yes, I'm mapping the code\ sub-folder namespace to the top level in flexlibs\__init__.py.
Okay, so the package_dir parameter isn't needed at all--I've just confirmed that that works.

[cdfarrow]

> py -3.8 -m build
* Creating venv isolated environment...
* Installing packages in isolated environment... (pythonnet >= 2.0.0, setuptools>=43.0.0, sphinx)
* Getting dependencies for sdist...
running egg_info
writing flexlibs.egg-info\PKG-INFO
writing dependency_links to flexlibs.egg-info\dependency_links.txt
writing requirements to flexlibs.egg-info\requires.txt
writing top-level names to flexlibs.egg-info\top_level.txt
package init file 'flexlibs\docs\__init__.py' not found (or not a regular file)
package init file 'flexlibs\tests\__init__.py' not found (or not a regular file)
package init file 'flexlibs\docs\flexlibsAPI\__init__.py' not found (or not a regular file)
package init file 'flexlibs\docs\flexlibsAPI\doctrees\__init__.py' not found (or not a regular file)
package init file 'flexlibs\docs\flexlibsAPI\html\__init__.py' not found (or not a regular file)
package init file 'flexlibs\docs\flexlibsAPI\_modules\__init__.py' not found (or not a regular file)
package init file 'flexlibs\docs\flexlibsAPI\_sources\__init__.py' not found (or not a regular file)
package init file 'flexlibs\docs\flexlibsAPI\_static\__init__.py' not found (or not a regular file)
package init file 'flexlibs\docs\flexlibsAPI\html\_modules\__init__.py' not found (or not a regular file)
package init file 'flexlibs\docs\flexlibsAPI\html\_sources\__init__.py' not found (or not a regular file)
package init file 'flexlibs\docs\flexlibsAPI\html\_static\__init__.py' not found (or not a regular file)
package init file 'flexlibs\docs\flexlibsAPI\html\_modules\flexlibs\__init__.py' not found (or not a regular file)
package init file 'flexlibs\docs\flexlibsAPI\_modules\flexlibs\__init__.py' not found (or not a regular file)
package init file 'flexlibs\docs\flexlibsAPI\_modules\flexlibs\code\__init__.py' not found (or not a regular file)
reading manifest file 'flexlibs.egg-info\SOURCES.txt'
reading manifest template 'MANIFEST.in'
adding license file 'LICENSE.txt'
writing manifest file 'flexlibs.egg-info\SOURCES.txt'
* Building sdist...

Can those "__init__.py not found" warnings be eliminated, or is this just expected behaviour?

[abravalheri]

Ok, I see now.

This will require a PR to pypa/distutils, but I think it is worthy the effort since distutils code preceedes PEP 420.

[cdfarrow]

Okay, sounds good.

[abravalheri]

pypa/distutils#151