Distribute to setuptools by singingwolfboy · Pull Request #8161 · openedx/openedx-platform

singingwolfboy · 2015-05-21T20:40:43Z

Follow-on PR from #7465.

Replaces distribute with setuptools
Updates MySQL-python to 1.2.5 for setuptools compatibility
Uses a forked version of NLTK 2 that does not require distribute

singingwolfboy · 2015-05-22T19:27:21Z

@feanil I have this branch running on setuptools.m.sandbox.edx.org, and it works -- and I verified that setuptools 16.0 is really installed. I imagine that we need to change some things about this PR before it can be merged, though.

sarina · 2015-05-26T19:18:06Z

@singingwolfboy does this replace #7465? What's going on?

singingwolfboy · 2015-05-30T07:24:34Z

@sarina yes, this is intended to replace #7465, although I don't think this PR is ready to be merged yet, since it depends on my fork of NLTK. To make this merge-ready, we should either make an NLTK fork and put it on the edx GitHub account, or upgrade to NLTK 3 (and deal with the backwards-incompatibilities). I'm waiting for some review & feedback from @feanil.

sarina · 2015-05-31T20:08:19Z

@singingwolfboy should #7465 be closed? Should an OSPR ticket be made for this, and be put for devops review?

singingwolfboy · 2015-06-01T13:14:29Z

I've altered OSPR-468 to point to this PR, instead of #7465.

benpatterson · 2015-06-02T12:36:38Z

@singingwolfboy could you give us an estimate of how much work it would take to rev up to NLTK3?

singingwolfboy · 2015-06-02T14:18:01Z

@benpatterson: The work to upgrade to NLTK 3 can be divided up into two parts. The first part is identifying all the different places that we use NLTK in our various different projects, finding backwards-incompatible usages, and rewriting them to be compatible with NLTK 3. (Ideally, using try/except blocks to make it compatible with either NLTK 2 or 3.) Then we have to test all those changes, of course. Based on a cursory search of the codebase in my text editor, it looks like we use NLTK in one or two places in edx-platform and several places in ease. There are also several references to NLTK in edx-ora2, but it doesn't look like edx-ora2 is using NLTK directly -- it's probably declaring NLTK as a dependency because ease uses it.

That's the first part. The second part is identifying any courses running on edx.org where the course author is running custom Python code as part of the course, and contacting those course authors to determine if they are using NLTK. If so, we'll also need to inform them that NLTK is going to be upgraded to version 3, point them to the documentation with the backwards-incompatibilities, and ideally show them how to write code that is compatible with either NLTK 2 or 3.

I'm guessing that the first part will take a few days of work, mostly spent on testing. I have no idea how to estimate how much time and effort is involved in the second part.

benpatterson · 2015-06-03T17:34:12Z

@singingwolfboy thanks for giving the details on it.

feanil · 2015-06-18T18:46:31Z

jenkins test this please

feanil · 2015-07-08T20:38:34Z

@singingwolfboy Brought up a sandbox and verified that at the end of the provision, it's the correct version. 🎆 I did a quick smoke test of the demo course as well and didn't see any issues. I think it's time. 👍

sarina · 2015-07-09T15:29:57Z

@singingwolfboy @feanil what needs to happen next?

feanil · 2015-07-09T15:37:52Z

@sarina we're trying to get ease and ora2 repos aligned before we can merge this.

sarina · 2015-07-09T15:43:02Z

Is there specific work we can point to on the JIRA ticket that indicates what it's blocked on?

'distribute' is the ancestor of 'setuptools', and many packages require setuptools to be installed. Upgrade of MySQL-python from 1.2.4 to 1.2.5 is required because of an incompatibility with setuptools>0.7.

So that we can use setuptools instead of distribute

feanil · 2015-07-09T17:21:48Z

This should be pinned instead of using -e

Out of curiosity to me, what does the -e do?

-e make pip install the dependency in "editable" mode: https://pip.pypa.io/en/stable/reference/pip_install.html#editable-installs I'm actually not sure what that means.

Distribute to setuptools

regisb · 2015-07-10T09:21:47Z

Thanks everyone for the great work!

singingwolfboy · 2015-07-13T14:07:12Z

@codref ansible no longer refers to post.txt as of two months ago: see https://github.com/edx/configuration/pull/1982 and https://github.com/edx/configuration/pull/1989. Can you make sure that you're updated to the latest version of the configuration repository, and let me know if you still get an error? If you do get an error, please post the traceback to a pastebin service (like Github Gist) and link to it, so that we can see what the problem is.

codref · 2015-07-14T09:08:41Z

Here you find the installation output with the error referring to a missing post.txt file.
The output came from a newly created ubuntu 12.04 virtual machine, I followed the steps described on the wiki (manual installation).

singingwolfboy · 2015-07-15T13:29:05Z

We had to partially revert this pull request in #8929, because there was an unexpected interaction between adding python-saml to our requirements (from #8140) and upgrading setuptools. In the interest of not delaying the weekly release to edx.org, we reverted setuptools and put distribute back in -- but we are planning to do the setuptools upgrade once we can figure out why python-saml was unable to build under setuptools.

singingwolfboy force-pushed the distribute-to-setuptools branch 2 times, most recently from 6e19ec4 to 4868a67 Compare May 22, 2015 18:52

singingwolfboy force-pushed the distribute-to-setuptools branch from 4868a67 to 1b95f62 Compare May 30, 2015 07:21

singingwolfboy mentioned this pull request Jun 1, 2015

Upgrade from distribute to modern version of setuptools #7465

Closed

singingwolfboy added the community manager review label Jun 1, 2015

openedx-webhooks added blocked by other work PR cannot be finished until other work is complete and removed community manager review labels Jun 1, 2015

openedx-webhooks added community manager review awaiting prioritization and removed blocked by other work PR cannot be finished until other work is complete community manager review labels Jun 2, 2015

sarina added the open-source-contribution PR author is not from Axim or 2U label Jun 2, 2015

singingwolfboy force-pushed the distribute-to-setuptools branch from 1b95f62 to d18b9a6 Compare June 3, 2015 17:36

singingwolfboy force-pushed the distribute-to-setuptools branch 2 times, most recently from 57fd57b to a635bfe Compare June 15, 2015 15:49

singingwolfboy force-pushed the distribute-to-setuptools branch from a635bfe to 0ed878b Compare June 30, 2015 13:45

openedx-webhooks added waiting on author PR author needs to resolve review requests, answer questions, fix tests, etc. and removed awaiting prioritization labels Jul 6, 2015

singingwolfboy force-pushed the distribute-to-setuptools branch 3 times, most recently from 33ec032 to 4bea63f Compare July 6, 2015 15:07

singingwolfboy force-pushed the distribute-to-setuptools branch from 4bea63f to 3388c4b Compare July 6, 2015 20:46

openedx-webhooks added awaiting prioritization and removed waiting on author PR author needs to resolve review requests, answer questions, fix tests, etc. labels Jul 7, 2015

singingwolfboy force-pushed the distribute-to-setuptools branch 2 times, most recently from 49620e9 to a673ce6 Compare July 9, 2015 16:00

regisb and others added 2 commits July 9, 2015 13:04

Upgrade distribute to the latest stable version of setuptools

95ebf7f

'distribute' is the ancestor of 'setuptools', and many packages require setuptools to be installed. Upgrade of MySQL-python from 1.2.4 to 1.2.5 is required because of an incompatibility with setuptools>0.7.

Use a forked version of NLTK

705fa69

So that we can use setuptools instead of distribute

singingwolfboy force-pushed the distribute-to-setuptools branch from a673ce6 to a31f685 Compare July 9, 2015 17:04

Use compatible versions of edx-ora2 and ease

a31f685

feanil reviewed Jul 9, 2015
View reviewed changes

singingwolfboy added 2 commits July 9, 2015 13:31

Don't install nltk in editable mode

28433e1

don't install ease in editable mode

8a6e310

openedx-webhooks added engineering review and removed awaiting prioritization labels Jul 9, 2015

singingwolfboy added a commit that referenced this pull request Jul 9, 2015

Merge pull request #8161 from edx/distribute-to-setuptools

da5ad4f

Distribute to setuptools

singingwolfboy merged commit da5ad4f into master Jul 9, 2015

singingwolfboy deleted the distribute-to-setuptools branch July 9, 2015 19:00

sarina mentioned this pull request Jul 9, 2015

Create post.txt #8869

Closed

singingwolfboy mentioned this pull request Jul 15, 2015

Re-upgrade to setuptools #8934

Merged

Uh oh!

Conversation

singingwolfboy commented May 21, 2015

Uh oh!

singingwolfboy commented May 22, 2015

Uh oh!

sarina commented May 26, 2015

Uh oh!

singingwolfboy commented May 30, 2015

Uh oh!

sarina commented May 31, 2015

Uh oh!

singingwolfboy commented Jun 1, 2015

Uh oh!

benpatterson commented Jun 2, 2015

Uh oh!

singingwolfboy commented Jun 2, 2015

Uh oh!

benpatterson commented Jun 3, 2015

Uh oh!

feanil commented Jun 18, 2015

Uh oh!

feanil commented Jul 8, 2015

Uh oh!

sarina commented Jul 9, 2015

Uh oh!

feanil commented Jul 9, 2015

Uh oh!

sarina commented Jul 9, 2015

Uh oh!

feanil Jul 9, 2015

Choose a reason for hiding this comment

Uh oh!

sarina Jul 9, 2015

Choose a reason for hiding this comment

Uh oh!

singingwolfboy Jul 9, 2015

Choose a reason for hiding this comment

Uh oh!

regisb commented Jul 10, 2015

Uh oh!

singingwolfboy commented Jul 13, 2015

Uh oh!

codref commented Jul 14, 2015

Uh oh!

singingwolfboy commented Jul 15, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants