# How to compare different models **URL:** \ **Category:** KNIME Analytics Platform\ **Created:** [January 22, 2018, 6:27pm UTC](https://forum.knime.com/t/how-to-compare-different-models/9947 "2018-01-22T18:27:15Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![zizoo](https://forum-cdn.knime.com/letter_avatar_proxy/v4/letter/z/2bfe46/32.png) [@zizoo](https://forum.knime.com/u/zizoo)\ **Post date:** [January 22, 2018, 6:27pm UTC](https://forum.knime.com/t/how-to-compare-different-models/9947/1 "2018-01-22T18:27:15Z")
Hello, I would like to compare different machine learning models ( SVM, random forests, Neural networks) for a multiclass database. I cannot use the ROC curve for this problem as I have more than 2 classes. I have from the scorer the confusion matrix and the Accuracy and the Cohen Kappa. There is also the F-measure, sensitivity, recall.. for every class. Can I say that the best model is the model with the highest accuracy or I should take in to account the other parameters. Is it appropriate to take the average of the F-measure, recall ..over the different classes? How can I measure the out-of-bag error for my model and how can I apply it for my models comparison? ---
**Author:** ![amartin](https://forum-cdn.knime.com/letter_avatar_proxy/v4/letter/a/7993a0/32.png) [@amartin](https://forum.knime.com/u/amartin)\ **Post date:** [January 29, 2018, 12:00pm UTC](https://forum.knime.com/t/how-to-compare-different-models/9947/2 "2018-01-29T12:00:52Z")
Hi zizoo,  The best measure for your models performance is heavily dependent on the problem. However, since you're dealing with more than 2 classes you have to keep in mind that only accuracy generalizes to more than 2 classes, although averaging the F-score, Recall and Precision measures can also be an option.  In case of both Random Forest and Tree Ensemble Learners the first output port gives the out-of-bag error estimates (more details in the node description), you can calculate the error using the Scorer node comparing the true class with the out-of-bag prediction. However, it is not possible to compute the out-of-bag error estimates for SVM and Neural Networks.    Best, Anna ---
**Author:** ![sundriyals](https://forum-cdn.knime.com/letter_avatar_proxy/v4/letter/s/76d3ee/32.png) [@sundriyals](https://forum.knime.com/u/sundriyals)\ **Post date:** [July 31, 2019, 4:02am UTC](https://forum.knime.com/t/how-to-compare-different-models/9947/3 "2019-07-31T04:02:48Z")
@amartin As you said > [@amartin](#): > > you can calculate the error using the Scorer node comparing the true class with the out-of-bag prediction In my case the scorer node shows overall accuracy = 0.933 Does this mean OOB = 1 - 0.933 = 0.067 ? Thanks -San ---
**Author:** ![beginner](https://forum-cdn.knime.com/letter_avatar_proxy/v4/letter/b/e495f1/32.png) [@beginner](https://forum.knime.com/u/beginner)\ **Post date:** [July 31, 2019, 4:49am UTC](https://forum.knime.com/t/how-to-compare-different-models/9947/4 "2019-07-31T04:49:34Z")
> [@zizoo](#): > > Can I say that the best model is the model with the highest accuracy or I should take in to account the other parameters. This depends on class distribution and the goal of your model. if you have approx. same number of observations per class and all classes are equally important, then yes accuracy is a good measure. However if one class is much more common or much rarer then no, it can be very misleading as especially in case of a rare class, the predictions could be very bad while not impacting accuracy much. Classic example: assume binary classification and 90% of observations are from one class, 10% from the other. A static model predicting everything as the majority class would hence have an accuracy of 90%. [Cohen’s kappa](https://en.wikipedia.org/wiki/Cohen%27s_kappa) is a measure that takes class distribution into account and can be used in conjunction with accuracy. However it still depends on your actual problem if it’s a usable metric and there is no clear agreement of what a good value is (highly depends on the problem) But the goal of your model is crucial accuracy might not matter much. For example in an HIV test you would want to avoid false-negatives at all costs even if it negatively affects your accuracy. ---
**Author:** ![ipazin](https://forum-cdn.knime.com/user_avatar/forum.knime.com/ipazin/32/806_2.png) [@ipazin](https://forum.knime.com/u/ipazin)\ **Post date:** [July 31, 2019, 10:43am UTC](https://forum.knime.com/t/how-to-compare-different-models/9947/5 "2019-07-31T10:43:37Z")
Hi there @sundriyals, > [@sundriyals](#): > > Does this mean OOB = 1 - 0.933 = 0.067 ? Don’t think so. I think what @amartin had in mind is to connect first output port of Random Forest and/or Tree Ensemble Learner nodes to Scorer node to get accuracy on out of bag estimations. Br, Ivan ---
**Author:** ![sundriyals](https://forum-cdn.knime.com/letter_avatar_proxy/v4/letter/s/76d3ee/32.png) [@sundriyals](https://forum.knime.com/u/sundriyals)\ **Post date:** [July 31, 2019, 12:41pm UTC](https://forum.knime.com/t/how-to-compare-different-models/9947/6 "2019-07-31T12:41:33Z")
Hi @ipazin Thanks for your response. Sorry, I am not expert in machine learning and its nomenclature. > [@ipazin](#): > > I think what @amartin had in mind is to connect first output port of Random Forest and/or Tree Ensemble Learner nodes to Scorer node to get accuracy on out of bag estimations I got the point made by @amartin ; But, as I understand, overall accuracy given by the scorer node means all the correctly predicted cases (including positives and negatives) out of total instances. For e.g. in following example ![image](https://forum-cdn.knime.com/uploads/default/original/2X/c/c424ede668a38ae941b0a482d8486a92130fc697.png) total cases = 1016+874+63+87 = 2040; correctly predicted = 1890; therefore accuracy = 0.926 (or 92.6% accuracy) Wrongly predicted data = 63+87, therefore error is 63+87/2040 = 0.0735 which obviously can also be obtained by deducting overall accuracy from 1 (or 100, if considering in percentage) i.e. 1 - 0.926 = 0.073 My question is for out of bag accuracy estimate, if we do the same thing (i.e. 1 - accuracy), do we call it OOB error or is it some other type of error? If not, how can we obtain OOB error from the RF learner node in Knime? ---
**Author:** ![ipazin](https://forum-cdn.knime.com/user_avatar/forum.knime.com/ipazin/32/806_2.png) [@ipazin](https://forum.knime.com/u/ipazin)\ **Post date:** [July 31, 2019, 2:09pm UTC](https://forum.knime.com/t/how-to-compare-different-models/9947/7 "2019-07-31T14:09:06Z")
Hi there, not sure myself 😃 Will ask for help. Br, Ivan ---
**Author:** ![nemad](https://forum-cdn.knime.com/user_avatar/forum.knime.com/nemad/32/53702_2.png) [@nemad](https://forum.knime.com/u/nemad)\ **Post date:** [July 31, 2019, 2:30pm UTC](https://forum.knime.com/t/how-to-compare-different-models/9947/8 "2019-07-31T14:30:50Z")
Hi @sundriyals, what you describe is the way to get the OOB error in KNIME. Cheers, Adrian ---
**Author:** ![sundriyals](https://forum-cdn.knime.com/letter_avatar_proxy/v4/letter/s/76d3ee/32.png) [@sundriyals](https://forum.knime.com/u/sundriyals)\ **Post date:** [July 31, 2019, 2:46pm UTC](https://forum.knime.com/t/how-to-compare-different-models/9947/9 "2019-07-31T14:46:37Z")
Hi, @nemad many thanks! ---
**Author:** ![DemandEngineer](https://forum-cdn.knime.com/user_avatar/forum.knime.com/demandengineer/32/26048_2.png) [@DemandEngineer](https://forum.knime.com/u/DemandEngineer)\ **Post date:** [October 21, 2020, 11:57pm UTC](https://forum.knime.com/t/how-to-compare-different-models/9947/10 "2020-10-21T23:57:42Z")
Hi All, I’m new… coming from Alteryx Designer and trying to get up to speed on Knime… I have a data set where I’m training 4 models (Logistic Regression, Decision Tree, Forest, and Boosted) with historical data to see which is best at predicting loan default risk base on data we have available at time of application. In Alteryx I have a Model comparison “node” that takes the Union node of all the Learners Nodes and the other input takes a Validation set to test against. In the output you get a report with Fit and Error Measure, Confusion Matrix for each Model, and combines all for each of these charts (Lift curve, Gain Chart, Precision and recall curve and ROC Curve). This let’s me see which model worked best and run my new data set through the winning model… or I could continue to tune the data set to see if the performance of the models change… until I’m ready to select a winner. Is there a similar node that does this in Knime? If no, how do I do this? ---
**Author:** ![beginner](https://forum-cdn.knime.com/letter_avatar_proxy/v4/letter/b/e495f1/32.png) [@beginner](https://forum.knime.com/u/beginner)\ **Post date:** [October 22, 2020, 8:11am UTC](https://forum.knime.com/t/how-to-compare-different-models/9947/11 "2020-10-22T08:11:56Z")
Alteryx and KNIME while superficial similar follow completely different philosophies. KNIME offers you the core building blocks to build your own algorithms. This means more “boilerplate” nodes but more flexibility. Alteryx offers ready.made “macros” with lots of hidden/blackboxy features but you get to do complicated things quickly with little flexibility. Hence no, KNIME does not have such a node out-of-the box. You can however on KNIME Hub look for the [AutoML component](https://hub.knime.com/knime/spaces/Examples/latest/00_Components/Automation/AutoML) and look in this forum for help/instructions regarding it. Having said that I don’t really like these black-boxy tools.Each model will most likley require a different way of evaluating it’s performance and thinking about which method to choose and why is a step to understand your data, model and their limitations. Example: > I have a data set where I’m training 4 models (Logistic Regression, Decision Tree, Forest, and Boosted) Why these models? Why not also SVM? Or KNN? Why 3 types of tree-based models? (xgboost will almost always beat the other ones and never loose “big”)? Black boxes prevent such type of reasoning. For example you shouldn’t use tree-based algos if you have a lot of one-hot encoded categorical data. /rant ---
**Author:** ![DemandEngineer](https://forum-cdn.knime.com/user_avatar/forum.knime.com/demandengineer/32/26048_2.png) [@DemandEngineer](https://forum.knime.com/u/DemandEngineer)\ **Post date:** [October 22, 2020, 7:24pm UTC](https://forum.knime.com/t/how-to-compare-different-models/9947/12 "2020-10-22T19:24:57Z")
Thanks for the suggestions. I’ll take a look at the components. Are components collections of nodes? like a meta node? Sorry I’m a newbie to Knime. This was part of an assignment which I assume was to help us see how each variation of tree-based models are different. In testing so far I have found the Random Forest model performs better on validation data it’s never seen before vs Boosted (at least in Alteryx)… but that may be my inexperience in configuring. ---
**Author:** ![mlauber71](https://forum-cdn.knime.com/user_avatar/forum.knime.com/mlauber71/32/73590_2.png) [@mlauber71](https://forum.knime.com/u/mlauber71)\ **Post date:** [October 22, 2020, 7:41pm UTC](https://forum.knime.com/t/how-to-compare-different-models/9947/13 "2020-10-22T19:41:57Z")
OK, there is the Binary Classification Inspector that would give you an idea about the model. [Binary Classification Inspector Example](https://kni.me/w/U0FdrCkxoS66e2LQ) I created two Meta nodes for binary and regression models that would run a certain set of parameters and create some graphics (you would need some R packages). Input would just be a solution column with the truth either as 0/1 string or the number and a “submission” which is the score from 0-1. You could collect the results from the statistic tables and see which model is best. But you might want to be careful since what is ‘best’ might depend on your business question and which cutoff you would come to use. The Meta node does suggest an optimal cutoff but you might want to include some costs (for misclassification). ![image](https://forum-cdn.knime.com/uploads/default/original/3X/f/9/f95d9a98a731b13a5a005fd44442e06e51e90b8a.png) The output is explained in more details here: > [@H2O.ai AutoML in KNIME for classification problems](https://forum.knime.com/t/h2o-ai-automl-in-knime-for-classification-problems/20923): > > H2O.ai Automl - a powerful auto-machine-learning framework wrapped with KNIME It features various models like Random Forest or XGBoost along with Deep Learning. It has wrappers for R and Python but also could be used from KNIME. The results will be written to a folder and the models will be stored in MOJO format to be used in KNIME (as well as on a Big Data cluster via Sparkling Water). One major parameter to set is the running time the model has to test various models and do some hyperparam… > [@H2O.ai AutoML in KNIME for regression problems](https://forum.knime.com/t/h2o-ai-automl-in-knime-for-regression-problems/20924): > > H2O.ai Automl - a powerful auto-machine-learning framework wrapped with KNIME It features various models like Random Forest or XGBoost along with Deep Learning. It has wrappers for R and Python but also could be used from KNIME. The results will be written to a folder and the models will be stored in MOJO format to be used in KNIME (as well as on a Big Data cluster via Sparkling Water). One major parameter to set is the running time the model has to test various models and do some hyperparam… If you want to explore the possibilities of KNIME and machine learning further I have created a meta collection > **[Machine Learning Meta Collection (with KNIME) – mlauber71](https://hub.knime.com/mlauber71/spaces/Public/_machine_learning_meta_collection~--zu7353LMafrYQ_/current-state)** > > Machine Learning Meta Collection (with KNIME) This meta collection is about machine learning. It contains links to some examples demonstrating several types of… ---
**Author:** ![ipazin](https://forum-cdn.knime.com/user_avatar/forum.knime.com/ipazin/32/806_2.png) [@ipazin](https://forum.knime.com/u/ipazin)\ **Post date:** [October 23, 2020, 4:41pm UTC](https://forum.knime.com/t/how-to-compare-different-models/9947/14 "2020-10-23T16:41:05Z")
Hello @DemandEngineer, > [@DemandEngineer](#): > > I’ll take a look at the components. Are components collections of nodes? like a meta node? Check this blog post about Components vs Metanodes: > **[Metanode or Component - What's the Difference? | KNIME](https://www.knime.com/blog/metanode-or-component/)** > > Let's help clarify the difference between metanodes and components. What's a metanode? What's a component? And when do you use what? The common goal: make order in a messy workflow Both metanodes and components are useful to clean up messy workflows.... And see here all-you-need-to-know-about-them guide: [https://docs.knime.com/2020-07/analytics\_platform\_components\_guide/index.html](https://docs.knime.com/2020-07/analytics_platform_components_guide/index.html) Br, Ivan ---
**Author:** ![system](https://forum-cdn.knime.com/uploads/default/original/1X/ab3ccf34482a0329361734a18199390177204f15.png) [@system](https://forum.knime.com/u/system)\ **Post date:** [June 23, 2023, 8:28pm UTC](https://forum.knime.com/t/how-to-compare-different-models/9947/15 "2023-06-23T20:28:13Z")