prediction in Spark

Hello,
I would like to know if it is possible to obtain class probabilities as a prediction in Apache Spark for the Spark MLlib models. The Spark Predictor (MLlib) produces a prediction but not class probabilities. Also, the Spark MLlib to PMML node does not support some Spark MLlib models.
Thank you.

Hi @datascience100

unfortunately for the “old” MLlib nodes it is not possible to obtain class probabilites. It is possible for the spark.ml based nodes however:

There are plans to revise more of the Spark learner nodes to use spark.ml, but currently no concrete date has been set for this.

2 Likes

Not a solution for this problem but you could use models from H2O.ai also on a Big Data cluster with Sparkling Water, like this: