Find outlier with neuronal networks

[schildjo]

Hi M42,

first of all I want to thank you for this nice explanation.

@M42

First of all, most people use the words “outlier detection” and “anomaly detection” as synonyms, but they are not. Outlier detection is a type of anomaly detection: Outlier detection is about finding outliers/anomalies within a given dataset (training data). Novelty detection is the other type of anomaly detection. Here we have training dataset of which we know that the training data is normal. Now we see some new data that has not been part of the training data and we want to know whether is like the training data (and thus normal) or whether it is not (it is an anomaly in regard to the training data).

I was a person who thought, that outlier/anomaly detection is the “same”
After reading your post its more clear to me.

@M42

So it is a little difficult to answer whether it is supervised or unsupervised and how much at which step. There are many methods, but it would be the first time I heard that using a feed-forward neural network is one of them - I am pretty sure there is some research about using neural network for anomaly detection, because there is research with neural network about everything , but it most likely is going to be a special one.

My idea was to look at the examples given by Knime. There is an example (04_Analytics/14_Deep_Learning/01_DL4J/03_Network_Example_Of_A_Simple_MLP) which, looks for me, similar to my project.

@M42

You see that novelty detection is kind-of-supervised, but with a single class: All the training data is labeled, but with the same label. …

Is this possible to do novelty detection in Knime with neuronal networks?
Can you tell me an example or a workflow or furhter reading?

I tested it with my workflow from above but maybe I miss something or doing something wrong. I used some trainingdata (about 1000 rows) with only “good” labels on the CSV-Reader at the top. In the second CSV-Reader I give some testdata where the “cpu value” is not normal (its the only attribute which is anormal at a time). So my hope was that choosing class column “label” in the Rprop MLP Learner i would get any plausible results. But this was not the case. So is my procedure completely wrong?

Doing it full supervised (labels good/bad) i got more plausible results (best ~90% accurancy). See the workflow below.

I’am a little bit lost doing the first steps in novelty detection.

Thanks for any help.

Johannes