# Duplicate Row filter

**URL:** <https://forum.knime.com/t/duplicate-row-filter/21645>\
**Category:** KNIME Analytics Platform\
**Created:** [March 10, 2020, 2:18pm UTC](https://forum.knime.com/t/duplicate-row-filter/21645 "2020-03-10T14:18:03Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![chandrika\_99](https://forum-cdn.knime.com/letter_avatar_proxy/v4/letter/c/da6949/32.png) [@chandrika\_99](https://forum.knime.com/u/chandrika_99)\
**Post date:** [March 10, 2020, 2:18pm UTC](https://forum.knime.com/t/duplicate-row-filter/21645/1 "2020-03-10T14:18:04Z")

</div>

Hi Team,

Hope you are well.

While using joiner(Left outer) I am getting duplicates so I used duplicate row filter to remove the duplicates but duplicates are not removed.  
So I tried on different column combinations to remove but still, the issue is there either I am getting more or I am getting less no. of line items.  
In excel 1 - Total no. of line items: 217803  
In excel 2 - Total no.of line items:154774  
After using Joiner (Left outer): 229606 (I want as per excel 1)  
Joiner based on Doc. No.  
I am getting duplicates  
So, I used Duplicate Row Filter: 215109  
On different column combinations different no. of line items.

Can you please suggest how to remove duplicates.

Thanks

---

<div class="post-metadata">

**Author:** ![pjacob](https://forum-cdn.knime.com/letter_avatar_proxy/v4/letter/p/71c47a/32.png) [@pjacob](https://forum.knime.com/u/pjacob)\
**Post date:** [March 10, 2020, 2:21pm UTC](https://forum.knime.com/t/duplicate-row-filter/21645/2 "2020-03-10T14:21:10Z")

</div>

look at the photo

> <https://stackoverflow.com/questions/45987384/using-and-in-an-inner-join>

---

<div class="post-metadata">

**Author:** ![ScottF](https://forum-cdn.knime.com/user_avatar/forum.knime.com/scottf/32/61316_2.png) [@ScottF](https://forum.knime.com/u/ScottF)\
**Post date:** [March 10, 2020, 6:41pm UTC](https://forum.knime.com/t/duplicate-row-filter/21645/3 "2020-03-10T18:41:49Z")

</div>

Hi @chandrika_99 -

Would you be able to provide a simple example workflow with toy datasets - maybe a few dozen rows per file? In this case it would help to be able to see what your intended inputs and outputs are, and what the duplication pattern is like.

---

<div class="post-metadata">

**Author:** ![mlauber71](https://forum-cdn.knime.com/user_avatar/forum.knime.com/mlauber71/32/73590_2.png) [@mlauber71](https://forum.knime.com/u/mlauber71)\
**Post date:** [March 10, 2020, 10:45pm UTC](https://forum.knime.com/t/duplicate-row-filter/21645/4 "2020-03-10T22:45:45Z")

</div>

I can add one more idea to use SQL and row\_number if you have an ID that would indicate a duplicate entry.

```auto
SELECT * FROM 

( SELECT * 
, row_number() over (partition BY `ID` ORDER BY `date` DESC) AS `rank_id`
FROM `default`.`data1`  
) t1
-- only keep the entry that will be 
-- left on top of a group of identical v_name
WHERE `t1`.`rank_id` = 1
;

```

> [@Remove rows with duplicate values](https://forum.knime.com/t/remove-rows-with-duplicate-values/11105/15):
>
> @serendipitytech I can add one more idea to remove duplicates and ensure unique IDs using SQL (Hive in this case) and row\_number() if you have to deal with a data base - or you just like using SQL script.

---

<div class="post-metadata">

**Author:** ![system](https://forum-cdn.knime.com/uploads/default/original/1X/ab3ccf34482a0329361734a18199390177204f15.png) [@system](https://forum.knime.com/u/system)\
**Post date:** [September 9, 2020, 10:45am UTC](https://forum.knime.com/t/duplicate-row-filter/21645/5 "2020-09-09T10:45:48Z")

</div>

This topic was automatically closed 182 days after the last reply. New replies are no longer allowed.