[BEAM-3761] Define cmp() in Python 3

[cclauss]

Signed-off-by: cclauss cclauss@bluewin.ch

Please add a meaningful description for your change here
Define cmp() function that was removed in Python 3. A more straight ahead approach to #4774 that should be easier to review. Fixes the following issues:

flake8 testing of https://github.com/apache/beam on Python 3.6.3

$ flake8 . --count --select=E901,E999,F821,F822,F823 --show-source --statistics

./sdks/python/apache_beam/io/vcfio_test.py:77:14: F821 undefined name 'cmp'
      return cmp(v1.end, v2.end)
             ^
./sdks/python/apache_beam/io/vcfio_test.py:78:12: F821 undefined name 'cmp'
    return cmp(v1.start, v2.start)
           ^
./sdks/python/apache_beam/io/vcfio_test.py:79:10: F821 undefined name 'cmp'
  return cmp(v1.reference_name, v2.reference_name)
         ^
./sdks/python/apache_beam/io/gcp/datastore/v1/helper.py:93:12: F821 undefined name 'cmp'
  result = cmp(p1.kind, p2.kind)
           ^
./sdks/python/apache_beam/io/gcp/datastore/v1/helper.py:101:12: F821 undefined name 'cmp'
    return cmp(p1.id, p2.id)
           ^
./sdks/python/apache_beam/io/gcp/datastore/v1/helper.py:106:10: F821 undefined name 'cmp'
  return cmp(p1.name, p2.name)
         ^
./sdks/python/apache_beam/transforms/window.py:195:12: F821 undefined name 'cmp'
    return cmp(self.end, other.end) or cmp(hash(self), hash(other))
           ^
./sdks/python/apache_beam/transforms/window.py:195:40: F821 undefined name 'cmp'
    return cmp(self.end, other.end) or cmp(hash(self), hash(other))
                                       ^
./sdks/python/apache_beam/transforms/window.py:250:14: F821 undefined name 'cmp'
      return cmp(type(self), type(other))
             ^
./sdks/python/apache_beam/transforms/window.py:251:12: F821 undefined name 'cmp'
    return cmp((self.value, self.timestamp), (other.value, other.timestamp))
           ^

---

Follow this checklist to help us incorporate your contribution quickly and easily:

• Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

It will help us expedite review of your Pull Request if you tag someone (e.g. @username) to look at it.

Post-Commit Tests Status (on master branch)

Lang	SDK	Apex	Dataflow	Flink	Gearpump	Samza	Spark
Go		---	---	---	---	---	---
Java
Python		---		---	---	---	---

[cclauss]

@aaltay @holdenk

[holdenk]

So I’d still rather remove them long term, but I agree in the short term hopefully this is easier to get a committer to sign off on.

That being said having 3 copies of the function isn’t great, how about moving it somewhere common?

[cclauss]

Agreed. When I thought about centralizing it but the three files are not near each other in the hierarchy (io, testing, transforms) but I suppose we could do something like from apache_beam.utils.compat import cmp It also stuck me as a bit weird (in legacy Python) to import a builtin from another file so that made the try/except block essential which in turn means that just one line per file is actually saved. I will make the change if that seems like the best approach.

[superbobry]

This one-liner has the same effect:

from past.bulitins import cmp

[cclauss]

@superbobry A perfect suggestion. Thanks much. Done.

[holdenk]

LGTM

[pabloem]

It seems that there were failures in Cython: https://builds.apache.org/job/beam_PreCommit_Python_Commit/166/console

[pabloem]

You can run this test locally via tox -e py27-cython from within sdks/python/

[superbobry]

I'm curious, why does Cython fail on Jenkins? I've tried locally with both Python 2.7 and 3.6 and Cython 0.28.1, and the following imports fine:

from past.builtins import unicode

print(unicode)

Could it be that py27-cython does not install future which is needed for the import to work? What do you think @pabloem?

[holdenk]

You could try updating tox.ini to explicitly mention that as a dep?

[cclauss]

The current approach is safer... Only redefine cmp() when it is necessary.

We only have 18 more months to get this done.

[holdenk]

No, I'm not suggesting you change your approach I'm talking about addressing the jenkins build issues?

[cclauss]

There are no Jenkins build issues using my approach ;-)

Besides, I think you already put futurize in all appropriate places in tox.ini

[holdenk]

Isn't https://builds.apache.org/job/beam_PreCommit_Python_Commit/166/console from your approach or am I looking at the wrong build logs?

[pabloem]

The build ran once more after d9d8689 was pushed. See: https://builds.apache.org/job/beam_PreCommit_Python_Commit/199/

[cclauss]

Yes. The commits were squashed into a single commit.

[holdenk]

Ah ok, sorry for my confusion then :)

[superbobry]

There are no Jenkins build issues using my approach ;-)

No, but your approach introduces boilerplate. I think this is worth fixing given that a) there seem to be no issue with importing from past on Cython, and b) a past import is no less safe than try-except NameError.

[cclauss]

OK. Can you please create a new PR that solves the problem and when it is checked in then I will gladly close this one. I have zero pride of ownership on these fixes... I merely want them resolved.

[superbobry]

Sure, will do! I actually have a PR open which addresses similar patterns in the code (see #5869).

[cclauss]

@charlesccychen Your review please?

[charlesccychen]

Run Python PostCommit

[charlesccychen]

Thanks! This LGTM. Will merge once tests pass.

[cclauss]

Can someone please help me to decipher what went wrong?

• https://builds.apache.org/job/beam_PostCommit_Python_Verify_PR/23/

[charlesccychen]

Strange, looks like it crashed / timed out in the middle.

[charlesccychen]

Run Python SDK PostCommit

[charlesccychen]

Run Python Dataflow ValidatesRunner

[cclauss]

I am even more confused.

[charlesccychen]

This looks fine, and the ValidatesRunner tests pass, so I'm going to merge even though Jenkins seems to buggy right now for the PostSubmit run.

[cclauss]

Thanks all!