[SPARK-3181][MLLIB]: Add Robust Regression Algorithm with Huber Estimator

[fjiang6]

Huber Robust Regression under spark/ml/regression
Unit Tests

[SparkQA]

Test build #40111 has finished for PR 8013 at commit 2f67e63.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class RobustRegression(override val uid: String)

[fjiang6]

@mengxr @dbtsai @srowen had RobustRegression in the same LinearRegression codebase as requested. And included the Unit Tests.

[dbtsai]

Still a lot of duplication. We're adding new features into LiR now, and it will be hard to maintain. Is it possible that you just add the objective function, and use Params to switch between different objective function? Thanks.

[SparkQA]

Test build #40222 has finished for PR 8013 at commit 96e38a7.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #40223 has finished for PR 8013 at commit 23e4c62.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[fjiang6]

@dbtsai ust added the objective function, and use Params to switch between different objective function. Thanks!

[dbtsai]

sharedParams.scala can not be edited directly. Please look at SharedParamsCodeGen.scala.

[dbtsai]

Also, make HasRobust as HasRobustRegression in SharedParamsCodeGen.scala.

[SparkQA]

Test build #40530 has finished for PR 8013 at commit 51e47dc.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class EqualNullSafe(attribute: String, value: Any) extends Filter

[fjiang6]

This class was not added by me. I didn't touch PySpark.

[SparkQA]

Test build #41422 has finished for PR 8013 at commit 1567635.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #41421 has finished for PR 8013 at commit 3bb5930.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #41423 has finished for PR 8013 at commit a04179b.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[feynmanliang]

I think it would be better to introduce a "costFunction" param which defaults to "LeastSquares" and pattern match in LinearRegression#L195 since that will force mutual exclusivity when more than two cost functions are possible

[SparkQA]

Test build #41662 has finished for PR 8013 at commit e447623.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[feynmanliang]

nit: This is not really an "Option", can we just make this say " Set whether to use robust Huber Cost Function"

[feynmanliang]

There is a lot of code repetition between this and #2096, perhaps you can make the mllib one wrap this?

[dbtsai]

Hello, robust tuning parameter k should not be a constant as you implemented.
In the paper, http://users.stat.umn.edu/~sandy/courses/8053/handouts/robust.pdf
k = 1.345σ where σ is the square error of current square loss. But this will be very expensive to compute the current square error of current square loss and then compute the huber loss, so I think it's reasonable to approximate the square error from previous weight.

[SparkQA]

Test build #49555 has finished for PR 8013 at commit 01601ee.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rxin]

Thanks for the pull request. I'm going through a list of pull requests to cut them down since the sheer number is breaking some of the tooling we have. Due to lack of activity on this pull request, I'm going to push a commit to close it. Feel free to reopen it or create a new one. We can also continue the discussion on the JIRA ticket.

@dbtsai there are a few pull requests that were waiting on your review. Can you revisit them even if they are closed?