Add support for PySpark MLLib models (#215)

* Create PySpark manager

* Fix date

* Remove docs

* Add java and shebangs

* Add pyspark model tests

* Add other model types

* Add pyspark example

* Fix paths in unit tests

* Cast posix path to str

* Fix paths

* Fix test

* Freeze dependencies (versions are in tests)

* Loosen flake8 requirements

* Loosen flake8 requirements

* Fix version in test

* Run black

* Loosen pandas requirement

Co-authored-by: Neal Lathia <neallathia@gmail.com>

Author: nlathia

CHANGELOG.md

@@ -6,6 +6,8 @@
 
 Added support for any s3-compatible object storage that you can access via [MinIO](https://min.io/): `ModelStore.from_minio()`.
 
+Added support for [Spark ML Lib](https://spark.apache.org/) models.
+
 You can move model stores that you created in one file system directory to another one (e.g., making it a mounted volume in a container) and `modelstore` will continue to work ([#209](https://github.com/operatorai/modelstore/pull/209), thanks [@hauks96](https://github.com/operatorai/modelstore/issues/189)).
 
 **🐛  Bug fixes & general updates**

README.md

@@ -58,6 +58,7 @@ pip install modelstore
 * [Scikit Learn](https://github.com/operatorai/modelstore/blob/main/examples/examples-by-ml-library/libraries/sklearn_example.py)
 * [Skorch](https://github.com/operatorai/modelstore/blob/main/examples/examples-by-ml-library/libraries/skorch_example.py)
 * [Shap](https://github.com/operatorai/modelstore/blob/main/examples/examples-by-ml-library/libraries/shap_example.py)
+* [Spark ML Lib](https://spark.apache.org/)
 * [Tensorflow](https://github.com/operatorai/modelstore/blob/main/examples/examples-by-ml-library/libraries/tensorflow_example.py)
 * [Transformers](https://github.com/operatorai/modelstore/blob/main/examples/examples-by-ml-library/libraries/transformers_example.py)
 * [XGBoost](https://github.com/operatorai/modelstore/blob/main/examples/examples-by-ml-library/libraries/xgboost_example.py)

bin/_build_library

@@ -1,3 +1,4 @@
+#!/bin/bash
 set -e
 
 VIRTUALENV_NAME=$(pyenv local)

bin/_cleanup

@@ -1,3 +1,4 @@
+#!/bin/bash
 echo "\n 🧼  Removing pycache files"
 find . | grep -E "(__pycache__|\.pyc|\.pyo$)" | xargs rm -rf
 

bin/_config

@@ -1,3 +1,4 @@
+#!/bin/bash
 #export PYTHON_VERSION=3.7.13
 export PYTHON_VERSION=3.8.12
 #export PYTHON_VERSION=3.9.9

bin/_release_prod

@@ -1,3 +1,4 @@
+#!/bin/bash
 set -e
 
 echo "\n ⏱  Uploading library to pypi..."

bin/_release_test

@@ -1,3 +1,4 @@
+#!/bin/bash
 set -e
 
 echo "\n ⏱  Uploading library to testpypi..."

bin/_remove_pyenv

@@ -1,3 +1,4 @@
+#!/bin/bash
 set -e
 
 source $(dirname $0)/_config

bin/_setup_brew

@@ -1,3 +1,4 @@
+#!/bin/bash
 set -e
 
 echo "\n ⏱  Running brew update and installing libomp"
@@ -16,4 +17,7 @@ export CXX=g++-11
 # https://github.com/dmlc/xgboost/issues/7039
 brew install rajivshah3/libomp-tap/libomp@11.1.0
 
+# To use pyspark
+brew install java
+
 echo "\n ✅  Done."

bin/_setup_pyenv

@@ -1,3 +1,4 @@
+#!/bin/bash
 set -e
 
 source $(dirname $0)/_config