Any advice to improve the performance of a classification model ?

Could you give me some good advices to improve the performance of a classification model (with three classes)? I’m using the Gradient Boosting algorithm but I can not reach an accuracy above 90%

Hello @Rubik,

this really depends on your setting.
Do you know that it is possible to reach a accuracy above 90%.
Is your dataset imbalanced? If it is, then accuracy is not a very good metric because if 90% of your class is of class A, a model predicting all samples to be of class A would also achieve 90% accuracy.

If you are certain that you can achieve more than 90%, then you can try to perform a parameter optimization on e.g. the number of trees in the Gradient Boosted Trees Learner.
For this you will need the KNIME Optimization Extension.
Note that I would recommend to combine the parameter optimization with cross validation to avoid overfitting on your testing set.

Best,

Adrian

1 Like

Thanks Nemad,
my data set is fairly balanced and I think it’s possible to achieve over 90% accuracy because I’m new to Knime and got 90% accuracy with a bruteforce parameter-tuning method. I set 105 in number of models, 0.2 in percentage of learning and 8 in number of levels. What do you think about it?

thanks in advance

8 levels is quite a lot for Gradient Boosted Trees I would recommend smaller values in the range 2-6.
You could also try the Random Forest Learner which is a good alternative to GBTs.
How large is your dataset? That is, how many columns and rows do you have in total and how do you split fot training and testing?

Best,

Adrian

My dataset has 901 rows and 21 columns. All features are numeric, apart from the label which is a string (1,0,-1). I have used linear correlation filter and low varaince filter in order to reduce the features (from 25 to 21). I have tried a few minutes ago to use less than 8 levels but the accuracy decreases by about two percentage points. Any other tips to maximize the accuracy? Maybe I shoud use a better classification algorithm but I am a beginner in data mining.

I will appreciate your help in every way, if you want you can write me directly on my email (antgio90@outlook.it)

Thanks

With such few rows you should definitely use cross validation to get a real idea of how good you are actually doing. You can find examples for this on the example server.
It is odd that your model requires so many levels especially given your small sample size.
The best classification algorithm really depends on your dataset but for most cases where you have tabular data Gradient Boosted Trees and Random Forests are typically among the most well performing algorithms.
Something I would also try is a Logistic Regression because if your problem is approximately linear, tree based models like decision trees, random forests and gbts have a hard time to approximate the linear decision boundary (unless it is axis parallel).

Anyhow, first implement cross validation to get a better idea on how well you are doing right now :wink:

I prefer to answer questions here in the forum because this way others facing the same challenges can come back to the threat in the future.

Best,

Adrian

1 Like

thanks for your reply,
I also tried the Logistic Regression but there are no improvements in accuracy. Surely I have implemented the cross validation (10 validations) and the accuracy is around 90%. Is there a better way to understand how good my classification system is?

Best,
Antonio

Ok, if you have your cross validation setup you can try multiple models and compare their performance in terms of accuracy.
It can also be interesting to see whether models make the same or different mistakes.
In the later case you can simply combine the predictions to arrive at a so called ensemble (note that random forests and GBTs are already ensembles but it is not uncommon to include them in even larger ensembles of different model types).
Building ensembles is perhaps the most popular way to improve model performance (just look at top competing solutions on platforms like kaggle) but they only make sense if the model actually vary in their predictions.

Best,

Adrian

1 Like

Thanks Adrian, you have been very helpful.
I have fused four classification models using Prediction Fusion Node: Gradient Boosted, Random Forest, AttributeSelected e Tree Ensemble. I reach 94.5% in accuracy, I feel satisfied, for now :slight_smile:

Hoping that my experience is useful to someone else, let me know if there is anything else to improve my model even further.

Have nice time,
Antonio

2 Likes

We had a similar discussion about metrics in multi class predictions here:

Besides accuracy you could use a LogLoss metric. That means that not only the outcome of prediction will be taken into account but also the ‘level of certainty’.

If you have two predictions from two models
A) 1=51%, 2=39%, 3=10% TRUTH=>1
B) 1=90%, 2=5%, 3=4% TRUTH=>1

both come up with the correct result (1). But prediction B) would be better as it predicts the correct answer with higher confidence and might be more robust. So you would prefer the model behind B). The LogLoss takes that into account.

I reworked the workflow from our late discussion and put in a comparison of 4 modelling approaches with the great H2O.ai nodes. In this case the Random Forest comes out on top but barely. Please be aware: there are dozens of settings to be played around with and with different settings GLM might carry the day. The models are just there to give an illustration. You could easily adopt more models.

The LogLoss is done in Python, since I did not found one within KNIME.

You can read about the KNIME scorer metric here:

And about LogLoss here:
http://wiki.fast.ai/index.php/Log_Loss

The workflow expects a string variable “Class” that should have values 1-6 (or more) but might have other values as well (I have not tested that). If you have a different set of classes it should adapt to that.

kn_example_multiclass.knar (403.3 KB)

You would also get an overview of what all models do in an exported Excel. With a matrix about true and false predictions.

As has been discussed earlier. Be careful with relying on 99.x% accuracy claims. The usefulness of a model very much depends on your business question, what the default outcome is and what cost/consequences a misclassification has. Eg. if 95% of your population are Class=1 and you would predict everyone to be Class=1 your Accuracy still might be nominally high but the model would still be useless.

4 Likes

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.