[SPARK-10973][ML][PYTHON] __gettitem__ method throws IndexError exception when we…

[zero323]

gettitem method throws IndexError exception when we try to access index after the last non-zero entry

from pyspark.mllib.linalg import Vectors
sv = Vectors.sparse(5, {1: 3})
sv[0]
## 0.0
sv[1]
## 3.0
sv[2]
## Traceback (most recent call last):
##   File "<stdin>", line 1, in <module>
##   File "/python/pyspark/mllib/linalg/__init__.py", line 734, in __getitem__
##     row_ind = inds[insert_index]
## IndexError: index out of bounds

[jkbradley]

ok to test

[jkbradley]

test this please

[jkbradley]

May as well use inds here for clarity since that's what is used elsewhere

[jkbradley]

LGTM

Ping @mengxr FYI. Also, which Spark versions are we patching?

[jkbradley]

@zero323 Can you please add tags "[ML] [PYTHON]" to the title of this PR?

[SparkQA]

Test build #1859 has finished for PR 9009 at commit d28a644.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[zero323]

@jkbradley Done.

[jkbradley]

Thanks! I'll merge this with master once tests pass.

Would you be able to send PRs against branch-1.3, branch-1.4, branch-1.5 in order to backport this to previous Spark versions? They can use the same JIRA number.

[SparkQA]

Test build #1862 has finished for PR 9009 at commit a1898ee.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[jkbradley]

merging with master

[zero323]

@jkbradley #9062, #9063, #9064

[zero323]

It should be possible to push this check before binary search: 8a695fe2c3344acd19279fcd539177426d436a02

[jkbradley]

@zero323 Good point, that's better for sure. Do you mind preparing a patch for 1.6 for that? (I don't think it's necessary to backport it everywhere.)

[zero323]

@jkbradley Sure, I can do it later this week. Should I open a new JIRA for that?

[jkbradley]

Yes, please. Thanks!

[zero323]

@jkbradley I've created a JIRA and opened a PR.