prediction in Spark

I would like to know if it is possible to obtain class probabilities as a prediction in Apache Spark for the Spark MLlib models. The Spark Predictor (MLlib) produces a prediction but not class probabilities. Also, the Spark MLlib to PMML node does not support some Spark MLlib models.
Thank you.

Hi @datascience100

unfortunately for the “old” MLlib nodes it is not possible to obtain class probabilites. It is possible for the based nodes however:

There are plans to revise more of the Spark learner nodes to use, but currently no concrete date has been set for this.


Not a solution for this problem but you could use models from also on a Big Data cluster with Sparkling Water, like this: