[SPARK-14938][ML] replace RDD.map with Dataset.as

[zhengruifeng]

What changes were proposed in this pull request?

Replace rdd with dataset in ML.
From:

dataset.select(col($(labelCol)).cast(DoubleType), f, w).rdd.map {
    case Row(label: Double, feature: Double, weight: Double) =>
        (label, feature, weight)
}

To:

dataset.select(col($(labelCol)).cast(DoubleType), f, w)
    .as[(Double, Double, Double)].rdd

How was this patch tested?

local build

[SparkQA]

Test build #57049 has finished for PR 12718 at commit d3df6d4.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds no public classes.

[rxin]

cc @jkbradley this seems like a good idea...

[zhengruifeng]

Now, Encoder for Vector is missing...

[SparkQA]

Test build #57105 has finished for PR 12718 at commit a69e1fd.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #57114 has finished for PR 12718 at commit ed90f24.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[zhengruifeng]

Vector, LabeledPoint, Instance and AFTPoint can not be used in Dataset.as now

[SparkQA]

Test build #57222 has finished for PR 12718 at commit e57332a.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[MLnick]

We should definitely have an encoder for vector udts... cc @dbtsai @viirya

[MLnick]

You need the col($(labelCol)).cast(DoubleType) since now all Predictors take any NumericType for labels, see #10355

[zhengruifeng]

Thanks, I will fix it

[viirya]

We have no implicit encoder for vector udt. But we can explicitly create it.

[MLnick]

@viirya yeah, sorry I meant we should create an encoder that can be used in ml... whether an implicit or explicit.

[viirya]

encoder now supports UDTs. You just need to declare one before you want to use it, since sql implicit does not include implicit ones for them.

[MLnick]

@viirya can you provide an example of how this works for use here in this PR?

[SparkQA]

Test build #57225 has finished for PR 12718 at commit b2101b2.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[viirya]

e.g., in BisectingKMeans, this patch changes

val data = dataset.select(col($(featuresCol))).rdd.map { case Row(point: Vector) => point }

to

val data = dataset.select(col($(featuresCol))).as[Vector].rdd

We can add:

implicit def vectorEncoder: Encoder[Vector] = ExpressionEncoder()

in the class.

[SparkQA]

Test build #57231 has finished for PR 12718 at commit 2378829.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #57235 has finished for PR 12718 at commit 530f037.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #57238 has finished for PR 12718 at commit 9584026.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #57245 has finished for PR 12718 at commit df5e917.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #57246 has finished for PR 12718 at commit 8863991.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #57247 has finished for PR 12718 at commit 9ef9a51.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[zhengruifeng]

@viirya Thanks. I updated this PR following your example.

[zhengruifeng]

@jkbradley @mengxr @jaceklaskowski The new Dataset.as API is appled to ML in this PR.

[jaceklaskowski]

Sorry, couldn't resist :) I'd change "label" to be a symbol 'label. Not very widely used, but think it deserves its place in the code.

[zhengruifeng]

@jaceklaskowski Do you mean change from col($(labelCol)).cast(DoubleType).as("label") to col($(labelCol)).cast(DoubleType).as(getDefault(labelCol).get) ?

[jaceklaskowski]

Other than the few places where you could use symbols not string literals LGTM. Excellent job! Thanks.

[zhengruifeng]

This PR is too old and have many conflict with current master. I will close it.