# Replace/Simplify row values to eliminate duplicates **URL:** \ **Category:** KNIME Analytics Platform\ **Created:** [May 26, 2020, 5:00pm UTC](https://forum.knime.com/t/replace-simplify-row-values-to-eliminate-duplicates/24121 "2020-05-26T17:00:22Z")\ **Posts on this page:** 1\ **Showing post:** 5
**Author:** ![mlauber71](https://forum-cdn.knime.com/user_avatar/forum.knime.com/mlauber71/32/73590_2.png) [@mlauber71](https://forum.knime.com/u/mlauber71)\ **Post date:** [May 27, 2020, 9:56am UTC](https://forum.knime.com/t/replace-simplify-row-values-to-eliminate-duplicates/24121/5 "2020-05-27T09:56:48Z")
I see these points: - in deduplication you can _aggregate_ towards a key. That means you will loose any other key that is not been aggregated to - if you want to preserve the information you will have to do something like \*\* aggregation functions like min, max, avg \*\* llists (list all the values in one cell) the group by function has such things \*\* separate variables that contain the information like \*\*\* preserved\_id\_1, preserved\_id\_2 - then you can choose to keep one line based on several criteria (that would contain the original structure and row integrity - like discussed in the links below). You would loose the information from the other lines \*\* if you want to preserve them you will have to aggregate them separately and then re-join via a preserved ID (preferably your key value) The reason why I (somtimes) like the solution with the row\_number forcing the reduction to a single unique line is that especially in very large datasets with a cascaded group by (over several fields) some duplicates might be lurking even if the providing database sweared that would not be possible (very common in Big Data environments). So if you want to be sure just eliminate a lone duplicate. But that very much depends on your business case. If you cannot loose any information at all you will have to resort to one of the methods mentioned above. > [@Remove rows with duplicate values](https://forum.knime.com/t/remove-rows-with-duplicate-values/11105/2): > > I set up a workflow to demonstrate how this could be done use group by to calculate how many duplicates there are (note: KNIME should introduce a generic COUNT(\*) function - I had to use a variable) if the count is larger then 1 it is a duplicate left join it back to the original data sort the data by ID and other variables if you want to keep one of the duplicates use the LAG column to identify which line is a 2nd, 3rd occurrence of a duplicate make a rule to keep just a single line of each … > [@Remove rows with duplicate values](https://forum.knime.com/t/remove-rows-with-duplicate-values/11105/4): > > I would like to go into my reasoning for deliberately focussing on an ID. Yes you can just remove duplicates by Group by or SELECT DISTINCT. In a lot of cases you have something like customer IDs or phone numbers. With DISTINCT what would happen in such a case phone\_no | adress | joined 555-1234 | main street 1 | 2008 555-1234 | main street 1C | 2009 555-1234 | Main Street 1 | 2008 you would still end up with 3 distinct cases and you would call/mail the customer three times, and what if yo… --- _[View the full topic](https://forum.knime.com/t/replace-simplify-row-values-to-eliminate-duplicates/24121)._