Compare multiple ML Models

[Brock_Tibert]

I have trained a few simple binary classification models. I see that I can use the Scorer node to generate tabular results of the output.

My question: Is there a node that we can use to compare multiple models at the same time? This would be the fit level statistics like accuracy, precision, recall across a variety of models.

For sake of comparison, Orange has a Test and Score node that provides a simple interface to compare multiple models. Link and Screenshot below to provide context for my question.

Orange Data Mining - undefined

[MartinDDDD]

I think I have seen something like that in the autoML component:

AutoML – knime

If I remember correctly (sorry cannot try right now), you can configure which models should be automatically trained etc. and then you get an interactive view with statistics / metrics…

This is from the Component description:

[Brock_Tibert]

Thanks Martin. This is for an MBA course that I am teaching this semester, and I am trying to avoid “automated” approaches to have the student build intuition around what is taking place.

[MartinDDDD]

I see - in that case using AutoML is definitely cheating ;-).

At least for me I have not come across a node that allows you to compare multiple models in the way you are looking for.

The way to achieve this without AutoML will require concatenating results of Scorer nodes of different models I think…

[mlauber71]

@Brock_Tibert I have set up this collection for binary classification

Heart Disease - Machine Learning Case - Comparing Algorithms – mlauber71

There also is one for regression

Regression - House Prices - Advanced Techniques including H2O.ai and Vtreat –...

In the folders you will also find automated data preparation and some multi class.

[Brock_Tibert]

Thanks for the suggestion!

[Brock_Tibert]

Thanks Mark. I am not interested in using AutoML for this class. Related to Martin’s idea, it seems you are collecting the results from each model for a wider comparison. Am I reading your diagram correctly?

If there isn’t a single node that does this, it’s ok. I can teach around a centralized way to compare. I asked because, as noted, Orange allows you to do this easily, as did RapidMiner/Altair, if I remember correctly. Not a big deal.

I appreciate the examples, which are definitely helpful for future ideas.

[mlauber71]

@Brock_Tibert

If there isn’t a single node that does this, it’s ok.

I think you can easily build a structure where you can compare the different approaches. I also have an unfinished project on my drive that would use advanced model in Python including data preparation. Will have to polish and publish that at some point.

Concerning Orange3 there is this.