Rewrite PythonInterpreter.

[astroshim]

What is this PR for?

I've been testing the python interpreter and I found at least 4 major issues in the current python interpreter.

1. not working streaming output.

• https://issues.apache.org/jira/browse/ZEPPELIN-2225

2. printed "..." when there is indent in the python code.

• https://issues.apache.org/jira/browse/ZEPPELIN-1929

3. very slow output of matplotlib

• https://issues.apache.org/jira/browse/ZEPPELIN-1894
• https://issues.apache.org/jira/browse/ZEPPELIN-1360

4. Unexpected output of matplotlib.

• https://issues.apache.org/jira/browse/ZEPPELIN-2107

so I changed python interpreter to use py4j based on pyspark interpreter and would be fixed above issues.
and I am going to recreate conda, docker for python interpreter ASAP.

What type of PR is it?

Bug Fix | Hot Fix | Refactoring

How should this be tested?

1. not working streaming output.

import time
for x in range(0, 5):
    print x
    time.sleep(1)

2. printed "..." when there is indent in the python code.

def fn():
    print("hi")

fn()

3. very slow output of matplotlib.

import matplotlib
import sys
import matplotlib.pyplot as plt
plt.plot([1,2,3])

4. Unexpected output of matplotlib.

import matplotlib.pyplot as plt
import matplotlib as mpl

# Make a figure and axes with dimensions as desired.
fig = plt.figure(figsize=(8, 3))
ax1 = fig.add_axes([0.05, 0.80, 0.9, 0.15])
ax2 = fig.add_axes([0.05, 0.475, 0.9, 0.15])
ax3 = fig.add_axes([0.05, 0.15, 0.9, 0.15])

# Set the colormap and norm to correspond to the data for which
# the colorbar will be used.
cmap = mpl.cm.cool
norm = mpl.colors.Normalize(vmin=5, vmax=10)

# ColorbarBase derives from ScalarMappable and puts a colorbar
# in a specified axes, so it has everything needed for a
# standalone colorbar.  There are many more kwargs, but the
# following gives a basic continuous colorbar with ticks
# and labels.
cb1 = mpl.colorbar.ColorbarBase(ax1, cmap=cmap,
                                norm=norm,
                                orientation='horizontal')
cb1.set_label('Some Units')

# The second example illustrates the use of a ListedColormap, a
# BoundaryNorm, and extended ends to show the "over" and "under"
# value colors.
cmap = mpl.colors.ListedColormap(['r', 'g', 'b', 'c'])
cmap.set_over('0.25')
cmap.set_under('0.75')

# If a ListedColormap is used, the length of the bounds array must be
# one greater than the length of the color list.  The bounds must be
# monotonically increasing.
bounds = [1, 2, 4, 7, 8]
norm = mpl.colors.BoundaryNorm(bounds, cmap.N)
cb2 = mpl.colorbar.ColorbarBase(ax2, cmap=cmap,
                                norm=norm,
                                # to use 'extend', you must
                                # specify two extra boundaries:
                                boundaries=[0] + bounds + [13],
                                extend='both',
                                ticks=bounds,  # optional
                                spacing='proportional',
                                orientation='horizontal')
cb2.set_label('Discrete intervals, some other units')

# The third example illustrates the use of custom length colorbar
# extensions, used on a colorbar with discrete intervals.
cmap = mpl.colors.ListedColormap([[0., .4, 1.], [0., .8, 1.],
                                  [1., .8, 0.], [1., .4, 0.]])
cmap.set_over((1., 0., 0.))
cmap.set_under((0., 0., 1.))

bounds = [-1., -.5, 0., .5, 1.]
norm = mpl.colors.BoundaryNorm(bounds, cmap.N)
cb3 = mpl.colorbar.ColorbarBase(ax3, cmap=cmap,
                                norm=norm,
                                boundaries=[-10] + bounds + [10],
                                extend='both',
                                # Make the length of each extension
                                # the same as the length of the
                                # interior colors:
                                extendfrac='auto',
                                ticks=bounds,
                                spacing='uniform',
                                orientation='horizontal')
cb3.set_label('Custom extension lengths, some other units')

plt.show()

Questions:

• Does the licenses files need update? no
• Is there breaking changes for older versions? no
• Does this needs documentation? no

[zjffdu]

@astroshim Have you checked #1495 which also rewrite the python interpreter ? And strongly +1 to rewrite python interpreter. And it would be better to consider extensibility for python interpreter. So that other places can reuse it such as pyspark interpreter.

[astroshim]

@zjffdu Thank you for your opinion!
Of course i checked #1495 and I totally agree of extensibility for python interpreter.
but As you can see, current python interpreter has lots of issues and i think these issues are really serious so I wanted it to fix them on current python interpreter but I couldn't.
It would be better if #1495 is done ASAP but I thought we also need to think about zeppelin users while implementing #1495.
Actually i think it would be better to rewrite python interpreter as python codes for simplicity.

[astroshim]

I just added conda interpreter.

[soralee]

What a nice fix! Let me test it out and give you a feedback!

[Leemoonsoo]

Tested and it works well.

I left some comments. Also binary file python/src/main/resources/python/py4j-0.9-src.zip shouldn't be in the source tree. Need to download this file in build time.

[Leemoonsoo]

logger.debug can be more appropriate.

[Leemoonsoo]

Shell we keep this PandasSql interpreter?

[Leemoonsoo]

Is there any reason removing the test?

[astroshim]

@Leemoonsoo thanks for reviewing.. Let me fix the issues you commented and add PandasSql.

[astroshim]

@Leemoonsoo fixed the issues you commented and added PandasSql.
and also added conda, pandasql interpreter test on PR description.
Could you help to review?

[Leemoonsoo]

Tested %python.conda and %python.pandasql and it works well.
Could you restore PythonDockerInterpreter.java and PythonDockerInterpreterTest.java for %python.docker as well?

To make it work, i think we need to let docker container mount directory that py4j exists using -v option, and then set PYTHONPATH env variable to load py4j library using -e option.

So this line

setPythonCommand("docker run -i --rm " + image + " python -iu");

can be updated to something like

String py4jDir = ZEPPELIN_HOME + "/interpreter/python";
String py4jFile = "py4j-{version}.zip";   // read directory py4jDir and get file starts with py4j
setPythonCommand("docker run -i --rm -v " + py4jDir + ":/zeppelin_python -e /zeppelin_python/" + py4jFile + " " + image + " python -iu");

[jongyoul]

I tested with the first two scripts and worked well.

[Leemoonsoo]

@astroshim made a pullrequest to this branch, to bringing docker support back . Please take a look and merge if you think it's okay.