RProp MLP Learner Roc Curve

KNIME Analytics Platform
1

I have a common question regarding the ROC Curve on the results of MultiLayerPerceptron Predictor after RProp MLP Learner Node (binary classification) It semms to me, that the ROC Curve is allways like for a “Ideal Model” (The curve is exactly on the ordinate and exactly on the “upper” abscissa).
I consider 2 simple Models with RProp MLP Learner /MultiLayerPerceptron Predictor. One is from the KNIME training:
“L4_ML_Introduction_to_Machine_Learning_Algorithms”. The other model is my “Test model” with completely different data and results, but regarding the ROC Curve - same picture…
I am grateful for aky advice.

2

Hi,
Welcome to the KNIME Forum! Do you mean that your area under the curve is always 1? This happens if the data is perfectly separable by the predictor and it makes no errors. Try adding a Target Shuffling node before your learning process starts. Does the ROC Curve change? It should be close to the black line denoting a random choice model now, otherwise something is wrong with your workflow.
Kind regards,
Alexander

3

Many Thanks!
Yes , area under the curve is always 0,999.
Should I apply Shuffling Node on my target variable (0/1) ?
Should I apply Shuffling Node on both: testing and training dataset?

4

Hi,
Yes, apply Target Shuffling on your target variable and do it before splitting into test and training. It shouldn’t matter much where you do it, but that’s the easiest way to check.
Kind regards,
Alexander

5

Hi, I did so, but no change. I attached my workflow.01_Simple_Neural_Network_solution.knwf (963.9 KB) You find the added Shuffling Node in the preprocessing mentanode (I had to).

I applied in addition Random Forest Learner/Predictor on the same training and testing data. Here looks the ROC Curve normal and with Target Shuffeling looks even better, not as expected close to the black line denoting a random choice model !?

So I have 2 questions now: wath is wrong with my R Prop Learner Model, so that the ROC Curve is aways looks “ideal” and why the Target Shuffeling makes the ROC Curve of the Random Fores better?

6

Hi,
in the ROC Curve’s configuration as Class column you have to select the actual class column, not the one with the predicted values, in your case “rank”. Then choose “Low” as Positive class value and P (rang=Low) as the only included column, as you already did. The way you have it set up right now compares the predicted probability to the predicted value, which of course always agrees.
Kind regards,
Alexander

3 Likes
7

Thank you very much.
Now the ROC Curve looks normal and with the Target Shuffling Node it goes close to the black line denoting a random choice model .

1 Like
8

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.