Unbalanced data - good practice

[ScottF]

Hi @zarniak -

This is a classic rare-event problem. Instead of deleting data from your “not-issue” group - which is almost certainly going to cause problems down the line - I would recommend some type of oversampling strategy for your “issue” group.

Here’s a thread from last month that describes the SMOTE node in KNIME, and how it could be implemented. Of course there are other ways to deal with this type of problem, but SMOTE might be a good place to start.

SMOTE for unbalanced data

Hello, I have an unballanced data of 7 classes. I read that people recommend SMOTE to balance it without the need to delete samples from oversampled classes. And, indeed, the accuracy of the model that I am generating with random forest or SVM increased significantly. But I feel that this improvement is fake and is kind of cheating. Actually, there 2 options in the SMOTE node ( oversample the whole dataset or oversample the minority). Without SMOTE, the acc = 0.34, with oversampling minority…

Here’s a more general article about unbalanced datasets in machine learning that is focused on implementation in R. It touches on SMOTE as well.

https://shiring.github.io/machine_learning/2017/04/02/unbalanced