Egg-info location incorrect for modified package_dir in setup.py

[selimb]

My Setup

My package lies inside an src directory. This decision was inspired by Ionel's blog post.
The parts of my config relevant to this issue are:
In setup.py

setup(
  packages=find_packages('src'),
  package_dir={'': 'src'},

tox.ini:

[testenv]
usedevelop=True

Problem

I noticed that develop-inst-nodeps was being run on every call to tox, even though the docs mention:

There is an optimization coded in to not bother re-running the command if $projectname.egg-info is newer than setup.py or setup.cfg.

This is because it only looks for the egg-info in the same directory as setup.py. However, it seems to me setup.py develop outputs egg-info to the package_dir, in my case src. Here is the relevant part of the tox code (in venv.py):

setup_py = setupdir.join('setup.py')
egg_info = setupdir.join('.'.join((name, 'egg-info')))

Suggestions

Of course, it could be I'm just missing something and that this is not the recommended approach. Assuming this is indeed a bug, here are my suggestions.

The best way would be to find out what package_dir is and look for egg-info in that directory. After a bit of digging it doesn't seem it can simply be extracted from the python setup.py command (like --name for instance). Then, one could either parse the setup.py file or the output of python setup.py develop, both of which I think are quite messy.

The other, much much simpler way, is to allow a package_dir key in tox.ini. This results in duplicated information...but how often is that setting likely to be changed?

I'd be up for submitting a PR for the latter approach.

[obestwalter]

edit: Sorry. Misread your post the first time around ... it's the other way around and you expect the same things as I would have, so there is definitely something strange going on. Will mark this as a bug for now.

Just to give you quick feedback ...

I did not read that blog post yet, but let me tell you that your posted setup settings look weird to me and I don't quite see the point. I will definitely look into that later, because I want to understand the reasoning behind that.

Of course, it could be I'm just missing something [...]

AFAIK the .egg-info is always located in the top level of the project - so next to where setup.py lies. If you expect it anywhere else you might get disappointed. This as well with the caveat of my current world view that has not been changed by that blog post yet.

[selimb]

Edit: just read your edit lol. Will leave this here for future reference anyway.

Thanks for the quick reply!

It's not the most conventional setup settings perhaps, but setuptools' documentation itself has an example of using src as the root of the source tree (it's also mentioned in several other places on that page, just Ctrl+F src).

Regarding the location of .egg-info, I refer to you this link, again from setuptools' documentation. Specifically:

This should normally be the root of your project’s source tree (which is not necessarily the same as your project directory; some projects use a src or lib subdirectory as the source root).

After reading that, I gather it is possible to modify it if calling setup.py egg_info explicitly with the egg-base option. For some reason you can't do this with the setup.py develop --egg-path -- it's not what it's for.

Anyhow, I'd personally refrain from calling setup.py with the more advanced and obscure options XD.

[RonnyPfannschmidt]

@obestwalter egg-info for develop is put into sys.path, if sys.path is a package dir like src, it goes there

[obestwalter]

I don't understand. sys.path is a list of paths to search for modules. Do you mean it is put into the first entry in sys.path? I guess I have to read up about this a bit. My experience is that this always ends up next to setup.py.

[RonnyPfannschmidt]

@obestwalter when you use package_dirs it does not unless you do a sdist as far as i understood

[obestwalter]

o.k. thanks for the clarification @RonnyPfannschmidt.

[selimb]

To expand on the sys.path comment. Here is the output of pip install --editable (note I'm currently on Cygwin but I don't think that matter).

$ python2.7 -m virtualenv ~/.venvs/srctest && source ~/.venvs/srctest/bin/activate

(srctests) $ ls mypackage
src  setup.py

(srctests) $ cat mypackage/setup.py
from setuptools import find_packages, setup

setup(
    name='in_src',
    packages=find_packages('src'),
    package_dir={'': 'src'},
    install_requires=[
        'requests',
        'click==5.0'
    ]
)


(srctest) $ python -c "import sys; print '\n'.join(sys.path)"

/home/selimb/.venvs/srctest/lib/python27.zip
/home/selimb/.venvs/srctest/lib/python2.7
/home/selimb/.venvs/srctest/lib/python2.7/plat-cygwin
/home/selimb/.venvs/srctest/lib/python2.7/lib-tk
/home/selimb/.venvs/srctest/lib/python2.7/lib-old
/home/selimb/.venvs/srctest/lib/python2.7/lib-dynload
/usr/lib/python2.7
/usr/lib/python2.7/plat-cygwin
/home/selimb/.venvs/srctest/lib/python2.7/site-packages

(srctest) $ pip install --editable mypackage
Obtaining file:///cygdrive/d/scripts/tmp/mypackage
Collecting requests (from in-src==0.0.0)
Collecting click==5.0 (from in-src==0.0.0)
Installing collected packages: requests, click, in-src
  Running setup.py develop for in-src
Successfully installed click-5.0 in-src requests-2.13.0

(srctest) $ python -c "import sys; print '\n'.join(sys.path)"

/home/selimb/.venvs/srctest/lib/python27.zip
/home/selimb/.venvs/srctest/lib/python2.7
/home/selimb/.venvs/srctest/lib/python2.7/plat-cygwin
/home/selimb/.venvs/srctest/lib/python2.7/lib-tk
/home/selimb/.venvs/srctest/lib/python2.7/lib-old
/home/selimb/.venvs/srctest/lib/python2.7/lib-dynload
/usr/lib/python2.7
/usr/lib/python2.7/plat-cygwin
/home/selimb/.venvs/srctest/lib/python2.7/site-packages
/cygdrive/d/scripts/tmp/mypackage/src


(srctest) $ tree mypackage
mypackage
├── setup.py
└── src
    ├── in_src
    └── in_src.egg-info

Notice how a line was appended to sys.path (which AFAIK it does by creating an egg-link file in site-packages, amongst other things).

I purposefully didn't do pip install --editable . to show that the current working directory doesn't matter.

[asottile]

Note that I also ran into something similar to this while doing some (admittedly awful) setup.py introspection. My solution (while slightly janky) worked pretty well for me and may benefit the tox project:

            out, srcdir = _call_setup_py(srcdir, _find_setup_py(), 'egg_info')
            for line in out.decode('UTF-8').splitlines():
                if line.startswith('writing ') and line.endswith('/PKG-INFO'):
                    _, pkg_info = line.split(' ')
                    egg_info_dir, _, _ = pkg_info.rpartition('/')
            requires_txt = os.path.join(srcdir, egg_info_dir, 'requires.txt')

The implementation of _call_setup_py is essentially subprocess.check_output((setup_py, *args)) (I also had to do some stuff with trying different executables and environments, but you probably won't need that bit).

rpartition can probably be replaced with os.path.split (lol a bit sloppy on my part).

I was of course searching for requires.txt -- I only skimmed the issue but I imagine the egg_info_dir is probably more of interest to tox

[obestwalter]

@selimb looks like a PR is on the horizon for this - great :)