Removing Duplicate rows from a CSV file.

[sgilmour]

I read somewhere but now I can’t find the Duplicate Variable Node
I am looking to take my CSV file and check for doubles by rows.
This way I can verify that there are no doubles in my csv file.
But I need to check the whole row since I have approximately 10 or 12 columns.
Is there any good example workflows on how to set this up.
Thanks for the help,
Scott

[elsamuel]

I read somewhere but now I can’t find the Duplicate Variable Node

Are you referring to the Duplicate Row Filter node?

[mlauber71]

I wrote this article about duplicates. Maybe this and the accompanying workflow can help.

School of duplicates - and how to deal with them

Dealing with duplicates is a constant theme with data scientist. And a lot of things can go wrong. The easiest ways to deal with them is SQL’s GROUP BY or DISTINCT. Just get rid of them and be done. But as this examples might demonstrate this might not always be the best option. Even if your data provider swears your combined IDs are unique especially in Big Data scenarios there might still be lurking some muddy duplicates and you should still be able to deal with them. And you should be able t…

[sgilmour]

Yes I am referring to the Duplicate Row Filter Node

[ipazin]

Hi @sgilmour,

did you find him? It is part of basic installation so should be visible in node repository…

Br,
Ivan

[sgilmour]

Yes I found the duplicate row filter node. I was just looking for some example workflows to give me a starting point on how to get started.

[ipazin]

Hi @sgilmour,

in that case you can find node on KNIME Hub and there you’ll see a list of workflows were it was used

Br,
Ivan

[sgilmour]

The example workflow helped me create the workflow. I was able to create the workflow and a sample file.3 Duplicate Rows Workflow.knwf (413.3 KB) . Just need to test it with my real excel sheet.