posargs configerror

[pytoxbot]

• Bitbucket: https://bitbucket.org/hpk42/tox/issue/150
• Originally reported by: ehopemorley
• Originally created at: 2014-01-31T15:42:29.801

With tox version 1.7.0 I get the following error:

tox.ConfigError: ConfigError: substitution key 'posargs' not found

Which appears to be caused by the following line in my tox.ini:

commands = python setup.py test --slowest --testr-args='{posargs}'

This worked fine with the previous version of tox i.e. 1.6.1

[pytoxbot]

Original comment by @chmouel

cool! thanks for fixing that. This has been a major painpoint for us

[pytoxbot]

Original comment by @hpk42

fix issue150: parse {posargs} more like we used to do it pre 1.7.0.
The 1.7.0 behaviour broke a lot of OpenStack projects.
See PR85 and the issue discussions for (far) more details, hopefully
resulting in a more refined behaviour in the 1.8 series.
And thanks to Clark Boylan for the PR.

→ <>

[pytoxbot]

Original comment by @chr0n1x

Any progress on this? I'm working with the OpenStack projects ;)

[pytoxbot]

Original comment by @msabramo

Yeah most tox.inis that I see either:

1. Don't use posargs at all
2. Use it posargs very simply: e.g.: py.test {posargs}

So it probably doesn't matter a whole lot to most people.

But OpenStack has a bazillion tox.inis that use the --testr-args='{posargs}' so they get impacted by this the most.

[pytoxbot]

Original comment by @hpk42

@offbyone do i understand correctly that this would make the open stack use case --testr-args='{posargs}' above work again? I am not sure because your proposed algorithm depends on how words are split, right?

@msabramo i tend to agree that if we don't find a solution soon, we should just revert the behaviour and aim for something better in 1.8. I don't think it's going to break things too much because not too many people do wild things with posargs from my experience.

[pytoxbot]

Original comment by @offbyone

I have a proposed solution that should address the main concerns:

pseudocode:

for w in command.words:
    if w == posargs:
        append posargs as split words
    elif posargs in w:
        nw = replace posargs with escaped string posargs' first value
        append nw
        append remaining posargs
    else
        append w

Proposed result matrix:

                w       ['w']           w w w       ['w', 'w', 'w']         "w w" w         ['w w', 'w']
cmd=[]          cmd=w   ['cmd=w']       cmd=w\ w\ w ['cmd=w w w']           cmd=w\ w w      ['cmd=w w', 'w']
cmd []          cmd w   ['cmd', 'w']    cmd w w w   ['cmd', 'w', 'w', 'w']  cmd 'w w' w     ['cmd', 'w w', 'w']
cmd '[]'        cmd w   ['cmd', 'w']    cmd 'w w w' ['cmd', 'w w w']        cmd "'w w' w"   ['cmd', "'w w' w"]

[pytoxbot]

Original comment by @offbyone

So, here's a summary of behaviour since 1.4. Details follow, but :

1.4-1.6

• tox called with 'quoted args':
  • command has 'quoted args': runs with 'quoted args'
  • command has unquoted args: runs with unquoted args
• tox called with unquoted args:
  • command has 'quoted args': runs with 'quoted args'
  • command has unquoted args: runs with unquoted args

1.7

• tox called with 'quoted args':
  • command has 'quoted args': runs with 'quoted args'
  • command has unquoted args: runs with 'quoted args'
• tox called with unquoted args:
  • command has 'quoted args': runs with unquoted args
  • command has unquoted args: runs with unquoted args

A more complex example, with mixed quoting:

tox-args.ini

[testenv]
whitelist_externals = ls
commands =
    ls -l '{posargs}'
    ls -l {posargs}

This setup exists:

touch "a file" a-file

tox 1.6

tox -c tox-args.ini -- 'a file' a-file

expectations

• command 0: expect failure
• command 1: expect success

actual

python runtests: commands[0] | ls -l a file a-file
ls: a file a-file: No such file or directory
ERROR: InvocationError: '/bin/ls -l a file a-file'
python runtests: commands[1] | ls -l a file a-file
ls: a: No such file or directory
ls: file: No such file or directory
-rw-r--r--  1 offby1  staff  0 Apr 14 09:28 a-file
ERROR: InvocationError: '/bin/ls -l a file a-file'

tox -c tox-args.ini -- "'a file' a-file"

expectations

• command 0: expect failure
• command 1: expect failure

actual

python runtests: commands[0] | ls -l a file a-file
ls: a: No such file or directory
ls: file a-file: No such file or directory
ERROR: InvocationError: '/bin/ls -l a file a-file'
python runtests: commands[1] | ls -l a file a-file
-rw-r--r--  1 offby1  staff  0 Apr 14 09:28 a file
-rw-r--r--  1 offby1  staff  0 Apr 14 09:28 a-file

tox -c tox-args.ini -- a\ file a-file

expectations

• command 0: expect failure
• command 1: expect success

actual

python runtests: commands[0] | ls -l a file a-file
ls: a file a-file: No such file or directory
ERROR: InvocationError: '/bin/ls -l a file a-file'
python runtests: commands[1] | ls -l a file a-file
ls: a: No such file or directory
ls: file: No such file or directory
-rw-r--r--  1 offby1  staff  0 Apr 14 09:28 a-file
ERROR: InvocationError: '/bin/ls -l a file a-file'

tox -c tox-args.ini -- 'a\ file a-file'

expectations

• command 0: expect failure
• command 1: expect failure

actual

python runtests: commands[0] | ls -l a\ file a-file
ls: a\ file a-file: No such file or directory
ERROR: InvocationError: '/bin/ls -l a\\ file a-file'
python runtests: commands[1] | ls -l a file a-file
-rw-r--r--  1 offby1  staff  0 Apr 14 09:28 a file
-rw-r--r--  1 offby1  staff  0 Apr 14 09:28 a-file

tox 1.7

tox -c tox-args.ini -- 'a file' a-file

expectations

• command 0: expect failure
• command 1: expect success

actual

python runtests: commands[0] | ls -l a file a-file
-rw-r--r--  1 offby1  staff  0 Apr 14 09:28 a file
-rw-r--r--  1 offby1  staff  0 Apr 14 09:28 a-file
python runtests: commands[1] | ls -l a file a-file
-rw-r--r--  1 offby1  staff  0 Apr 14 09:28 a file
-rw-r--r--  1 offby1  staff  0 Apr 14 09:28 a-file

tox -c tox-args.ini -- "'a file' a-file"

expectations

• command 0: expect failure
• command 1: expect failure

actual

python runtests: commands[0] | ls -l 'a file' a-file
ls: 'a file' a-file: No such file or directory
ERROR: InvocationError: "/bin/ls -l 'a file' a-file"
python runtests: commands[1] | ls -l 'a file' a-file
ls: 'a file' a-file: No such file or directory
ERROR: InvocationError: "/bin/ls -l 'a file' a-file"

tox -c tox-args.ini -- a\ file a-file

expectations

• command 0: expect failure
• command 1: expect success

actual

python runtests: commands[0] | ls -l a file a-file
-rw-r--r--  1 offby1  staff  0 Apr 14 09:28 a file
-rw-r--r--  1 offby1  staff  0 Apr 14 09:28 a-file
python runtests: commands[1] | ls -l a file a-file
-rw-r--r--  1 offby1  staff  0 Apr 14 09:28 a file
-rw-r--r--  1 offby1  staff  0 Apr 14 09:28 a-file

tox -c tox-args.ini -- 'a\ file a-file'

expectations

• command 0: expect failure
• command 1: expect failure

actual

python runtests: commands[0] | ls -l a\ file a-file
ls: a\ file a-file: No such file or directory
ERROR: InvocationError: '/bin/ls -l a\\ file a-file'
python runtests: commands[1] | ls -l a\ file a-file
ls: a\ file a-file: No such file or directory
ERROR: InvocationError: '/bin/ls -l a\\ file a-file'

[pytoxbot]

Original comment by @msabramo

Here's another example that illustrates an earlier point that @cboylan made using the ls command.

make

# Makefile.ls_example

POSARGS = A file with multiple words

test:
    ls -l '$(POSARGS)'
    echo
    ls -l $(POSARGS)

$ make -f Makefile.ls_example
ls -l 'A file with multiple words'
-rw-r--r--  1 marca  staff  0 Apr  3 19:44 A file with multiple words
echo

ls -l A file with multiple words
ls: A: No such file or directory
ls: file: No such file or directory
ls: multiple: No such file or directory
ls: with: No such file or directory
ls: words: No such file or directory
make: *** [test] Error 1

The point here is that ls -l '$(POSARGS)' (quoted) and ls -l $(POSARGS) (not quoted) do different things.

tox

 # ls_example.ini

[testenv]
whitelist_externals = ls
commands =
    ls -l '{posargs}'
    ls -l {posargs}

$ tox -c ls_example.ini -- "A file with multiple words"
GLOB sdist-make: /Users/marca/dev/hg-repos/tox/setup.py
python inst-nodeps: /Users/marca/dev/hg-repos/tox/.tox/dist/tox-1.7.1.zip
python runtests: PYTHONHASHSEED='2518930073'
python runtests: commands[0] | ls -l A file with multiple words
-rw-r--r--  1 marca  staff  0 Apr  3 19:44 A file with multiple words
python runtests: commands[1] | ls -l A file with multiple words
-rw-r--r--  1 marca  staff  0 Apr  3 19:44 A file with multiple words

Here, the quoted form and non-quoted form do exactly the same thing, so expressiveness has been lost because the substitution eats the quotes.

I personally would rather have '{posargs}' have the old behavior again. It seems like it was probably that way a lot longer than it was the new way, which only surfaced in tox 1.7.0. Thus, there are likely to be more folks relying on the old semantics than the new semantics. I have seen that using --testr-args='{posargs}' is very common in a lot of OpenStack projects (the world from which @cboylan and @ehopemorley are coming). OpenStack has a lot of projects and a lot of tox.inis I think that uses this.

See:

• https://bugs.launchpad.net/openstack-ci/+bug/1274135
• http://www.gossamer-threads.com/lists/openstack/dev/35955
• http://openstack.10931.n7.nabble.com/Openstack-dev-All-tox-1-7-0-error-while-running-tests-td31850.html
• http://osdir.com/ml/openstack-dev/2014-02/msg01152.html

So I would say that it would be good to have '{posargs}' have the old behavior again, even if it wasn't the behavior intended by the author, because the old behavior is useful and is being relied upon. If folks want a different semantics, put that in a new variable ({posargs-list}?) or add a setting to switch on those semantics. But I don't really know what the quote-eating behavior is useful for -- I haven't seen a motivating example for that.

[pytoxbot]

Original comment by @msabramo

@hpk42 mentioned bash on IRC as a possible model to follow since tox is a command executor.

It just occurred to me that perhaps make is an even closer match, as a tox.ini is extremely similar to a Makefile in purpose and function. Bash has a wider scope than make, because it is used interactively and in scripts. Thus it has a lot of complex substitutions and even a full programming language. By contrast, make is used only for Makefiles and is much simpler, just giving a simple structure and simple variable substitutions.

For example, take this Makefile:

# Makefile

POSARGS = --failing --parallel

test:
    python setup.py test --slowest --testr-args='$(POSARGS)'

$ make
python setup.py test --slowest --testr-args='--failing --parallel'

$(POSARGS) is replaced with the value of the variable. The quotes surrounding it are not touched.

I think tox should function the same way but it doesn't. Note below that the quotes are stripped out:

# testr.ini

[testenv]
commands =
    python setup.py test --slowest --testr-args '{posargs}'

$ tox -c testr.ini -- --failing --parallel
GLOB sdist-make: /Users/marca/dev/hg-repos/tox/setup.py
python inst-nodeps: /Users/marca/dev/hg-repos/tox/.tox/dist/tox-1.7.1.zip
python runtests: PYTHONHASHSEED='614422397'
python runtests: commands[0] | python setup.py test --slowest --testr-args --failing --parallel
...

Two problems in tox:

1. I had to separate --testr-args and {posargs} with a space, because it doesn't work at all with an equals sign. That shouldn't be necessary.
2. Even though I put single quotes around {posargs}, they were stripped out, effectively making '{posargs}' the same as {posargs}.

[pytoxbot]

Original comment by @msabramo

Personally, I think the most intuitive behavior (how I expected it to work and was surprised when it didn't work this way) was to do simple, dumb string substitution -- {posargs} gets replaced with your args, space-separated, plan and simple. This is intuitive and easy to understand and reason about. Quotes should never be eaten. As @jesusaurus put it, the substitution should not be context-sensitive.

I gravitate towards simple string substitution with no quote eating, along the lines of what @cboylan is suggesting. To me that is easy to implement without bugs and understandable. Trying to emulate bash or do anything complex I think is going to be fraught with peril and I don't see why it's necessary.

[pytoxbot]

Original comment by @cboylan

@hpk42 Something like {posargs-string} would restore the old behavior if I rewrite all of my tox.ini files. Ideally I wouldn't need to do that, but if tox is committed to not introducing a second backward incompatible change in as many releases I will have to live with that.

@offbyone I don't think it is hard if you don't overthink it. The variable substitutions should be regular (IMO all of them should be not just special ones) then allow shlex to parse the resulting string. This makes the behavior consistent and predictable.

I'm not sure I have time to update my pull request to support {posargs-string} prior to Friday (I think I would need to rewrite much of it to deal with the special case instead of the current behavior of that PR which is to apply the same rules to every variable substitution).

[pytoxbot]

Original comment by @offbyone

That's the thing; the handling of whitespace really depends on context. In a shell command, escaping can take several forms -- single-quoting, double-quoting, -escaping -- to say nothing of other shells (I assume that Windows shells will do something different).

How big of a change is someone willing to write in here? Because to handle the various whitespace scenarios we really need to have support for these:

Given the command tox -- foo bar baz:

ls -l {posargs}   --> ls -l foo bar baz
ls -l '{posargs}'  --> ls -l 'foo bar baz' 
ls -l "{posargs}" --> ls -l "foo bar baz"

Given the command tox -- "foo bar" baz:

ls -l {posargs}   --> ls -l "foo bar" baz

# oooohkay... here's where I run out of ideas.

ls -l '{posargs}'  --> ls -l '"foo bar" baz' ?
ls -l "{posargs}" --> ls -l "foo\ bar baz"  ?

The takeaway, shell quoting is hard. I don't think a one-size-fits-all solution will work.

[pytoxbot]

Original comment by @hpk42

@cboylan @offbyone i'd like to resolve this issue, if possible, ahead of 1.7.1 which i'd like to do on friday. Any chance we can conclude on a course of action?

[pytoxbot]

Original comment by @hpk42

Hum, good point regarding escaping with {posargs-string}. Is escaping spaces the single args with \ a good idea in this case?