Tox shouldn't call pip directly to avoid shebang limitations

[pytoxbot]

• Bitbucket: https://bitbucket.org/hpk42/tox/issue/66
• Originally reported by: @tabo
• Originally created at: 2012-11-23T19:25:35.269

When the python interpreter for an environment is in a deep directory hierarchy (in my case due to a jenkins matrix with 4 axis), tox will make a pip call like:

#!sh

../bin/pip

The shebang in this file has the full path for the python interpreter in the virtualenv. In one of my failing tests it was:

#!sh

#!/home/tabo/jenkins/workspace/tbtest3/TOX_DB/postgres/TOX_DJANGO/dj14/TOX_PYTHON/py27/os/linux/.tox/py27-dj14-postgres/bin/python2.7

With a path length of 131 that exceeds linux's limitation of 127. Patching tox to call $PYTHON_BIN_PATH $PIP_BIN_PATH instead of just $PIP_BIN_PATH solves the problem.

[pytoxbot]

Original comment by @hpk42

addresses issue66: add --workdir option to override where tox stores its ".tox" directory
to "pip freeze" to obtain the list of installed packages.
Thanks Ted Shaw, Holger Krekel.
close issue66: add documentation to jenkins page on how to avoid
"too long shebang" lines when calling pip from tox. Note that we
can not use "python -m pip install X" by default because the latter
adds the CWD and pip will think X is installed if it is there.
"pip install X" does not do that.

(experimental) New feature: When a search for a config file fails, tox tries loading

→ <<cset 11c13b4313a4>>

[pytoxbot]

Original comment by @hpk42

addresses issue66: add --workdir option to override where tox stores its ".tox" directory
to "pip freeze" to obtain the list of installed packages.
Thanks Ted Shaw, Holger Krekel.
close issue66: add documentation to jenkins page on how to avoid
"too long shebang" lines when calling pip from tox. Note that we
can not use "python -m pip install X" by default because the latter
adds the CWD and pip will think X is installed if it is there.
"pip install X" does not do that.

(experimental) New feature: When a search for a config file fails, tox tries loading

→ <<cset 11c13b4313a4>>

[pytoxbot]

Original comment by @hpk42

So it turns out defaulting to "python -m pip" is a breaking change we cannot do. The problem is that this invocation adds the current working directory and this makes pip think a package is installed if it is in the invocation current working directory. You can change "install_command" to use "python -m pip" if that works for you (e.g. if you don't have packages in your root dir) or, probably a better idea, use the new "--workdir" option to have a tox working directory which is not deeply nested. The latter method has the added benefit that also all other invocations of scripts in a virtualenv will work. Even if we made "python -m pip" work you would still not be able to invoke other installed scripts in venvs that have too long path names.

[pytoxbot]

Original comment by @hpk42

fix issue66 by introducing using default "python -m pip"
and introducing list_dependencies_command. (Ted, Holger)

→ <<cset 73d3b0cabef0>>

[pytoxbot]

Original comment by @hawkowl

Yeah, I'm hitting this too on Twisted, as we try and move tox-first, including on the Windows builders.

With the install_command workaround, it works better, but then fails right after with:

GLOB sdist-make: /Users/red/code/tw spc/setup.py
py27-tests create: /Users/red/code/tw spc/build/py27-tests
py27-tests installdeps: cryptography >= 0.9, pyopenssl >= 0.13, service_identity, idna >= 0.6, pyserial, python-subunit, pycrypto, appdirs, soappy
py27-tests inst: /Users/red/code/tw spc/build/dist/Twisted-16.1.1.zip
Traceback (most recent call last):
  File "/Users/red/venvs/commands/bin/tox", line 11, in <module>
    sys.exit(cmdline())
  File "/Users/red/venvs/commands/lib/python2.7/site-packages/tox/session.py", line 39, in main
    retcode = Session(config).runcommand()
  File "/Users/red/venvs/commands/lib/python2.7/site-packages/tox/session.py", line 373, in runcommand
    return self.subcommand_test()
  File "/Users/red/venvs/commands/lib/python2.7/site-packages/tox/session.py", line 540, in subcommand_test
    output = py.process.cmdexec("%s freeze" % (pip))
  File "/Users/red/venvs/commands/lib/python2.7/site-packages/py/_process/cmdexec.py", line 28, in cmdexec
    raise ExecutionFailed(status, status, cmd, out, err)
py.process.cmdexec.Error: ExecutionFailed: 127  /Users/red/code/tw spc/build/py27-tests/bin/pip freeze
/bin/sh: /Users/red/code/tw: No such file or directory

I believe https://bitbucket.org/hpk42/tox/pull-requests/172/do-not-call-pip-directly-trying-to-fix/diff should fix this issue.

[pytoxbot]

Original comment by gralamin

This is still an issue I hit (running tox 1.7.2). Has this been fixed in any newer versions and missed being closed, or is it still an open defect?

Thanks!

[pytoxbot]

Original comment by @hpk42

thanks for your perseverance!

[pytoxbot]

Original comment by @tabo

Plan to do that tonight. I'm still cleaning up the mess I did including prints in the stdlib in that server while debugging this :)

[pytoxbot]

Original comment by @hpk42

could you prepare a patch including adapting tests?

[pawamoy]

Just referencing other related issues in virtualenv and pip:

• Shebang length exceeded in pip executable pypa/virtualenv#596
• Shebang length exceeded in pip executable pypa/pip#1773

I'm still experiencing this problem when using tox through tox itself with long testenv names!
I'll try --workdir option and give some feedback.

[jsharpe]

For reference using install_command is not sufficient for this as there is a later call to pip freeze which is not customisable so for this to work we'd need to be able to also override the pip freeze call too.