# Problems deleting duplicate Records **URL:** \ **Category:** KNIME Analytics Platform\ **Created:** [March 3, 2026, 12:21pm UTC](https://forum.knime.com/t/problems-deleting-duplicate-records/91536 "2026-03-03T12:21:29Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![HolgerKleinert](https://forum-cdn.knime.com/user_avatar/forum.knime.com/holgerkleinert/32/85044_2.png) [@HolgerKleinert](https://forum.knime.com/u/HolgerKleinert)\ **Post date:** [March 3, 2026, 12:21pm UTC](https://forum.knime.com/t/problems-deleting-duplicate-records/91536/1 "2026-03-03T12:21:29Z")
![image](https://forum-cdn.knime.com/uploads/default/original/3X/7/b/7b5cc1dc90d273e237bb190476b7f3073972bd0a.png) Hello everyone, I’ve already learned quite a bit about Knime Analytics (version 5.4.5), but now I’m at my wit’s end and I can’t find any information that helps me. I want to delete duplicate records that have the same value in the “BELEGNUMMER” column, but different values ​​in the “CPU\_DATE” and “CPU\_TIME” columns. It doesn’t matter which duplicate record remains. For other deletions, I use the “Rule bases Row Filter” where I have direct information about what should be deleted. This, for example, doesn’t work with the “Rule bases Row Filter”: $BELEGNUMMER$ = $BELEGNUMMER$ AND NOT $CPU\_DATE$ = $CPU\_DATE$ AND NOT $CPU\_TIME$ = $CPU\_TIME$ =\> TRUE because, like my other attempts, it returns an empty table. Can anyone help me with this? Thanks and best regards, Holger ---
**Author:** ![MartinDDDD](https://forum-cdn.knime.com/user_avatar/forum.knime.com/martindddd/32/69988_2.png) [@MartinDDDD](https://forum.knime.com/u/MartinDDDD)\ **Post date:** [March 3, 2026, 12:44pm UTC](https://forum.knime.com/t/problems-deleting-duplicate-records/91536/2 "2026-03-03T12:44:15Z")
Hey there and welcome to the community, I think with the rule-based row filter you will struggle to achieve your desired outcome as to the best of my knowledge you cannot reference a previous or next row. And for this task you also want to avoid looping. If I get it right, you want to have an output which only contains each BELEGNUMMER once and you do not care about which CPU\_DATE or CPU\_TIME is kept for the same BELEGNUMMER. So if you start with this table (dummy data…) ![image](https://forum-cdn.knime.com/uploads/default/original/3X/c/2/c24362db72895fad12dd4e0b26da2454db867732.png) … and this outcome is acceptable: ![image](https://forum-cdn.knime.com/uploads/default/original/3X/c/1/c15986f2961e7f7a11c18b3f71f7b6b7be3c9abc.png) Then the solution is the Duplicate Row Filter Node: ![image](https://forum-cdn.knime.com/uploads/default/original/3X/1/c/1c3aff6146eb12a56804da587e7af0ac0941ddd3.png) In the bottom part you have options to define which row is kept if there are duplicates: ![image](https://forum-cdn.knime.com/uploads/default/original/3X/e/4/e4d913dabbe250c00b3d116bccb074107b519cdb.png) ---
**Author:** ![mlauber71](https://forum-cdn.knime.com/user_avatar/forum.knime.com/mlauber71/32/73590_2.png) [@mlauber71](https://forum.knime.com/u/mlauber71)\ **Post date:** [March 3, 2026, 12:47pm UTC](https://forum.knime.com/t/problems-deleting-duplicate-records/91536/3 "2026-03-03T12:47:00Z")
@HolgerKleinert you first will have to define the exact rule that will define a duplicate. And then you can see if the Duplicate Row Filter can help or if you might want to use WINDOW functions with the help of H2 database. > **[DB SQL - H2 School of duplicates - and how to deal with them – mlauber71](https://hub.knime.com/mlauber71/spaces/Examples/DB%20SQL%20-%20H2%20School%20of%20duplicates%20-%20and%20how%20to%20deal%20with%20them~liNhhbAUOVWOTmZb/current-state)** > > School of duplicates - and how to deal with them - H2 version Dealing with duplicates is a constant theme with data scientist. And a lot of things can go wrong… Also you might take a look at this approach by @takbb > [@Duplicate Filter with more conditions](https://forum.knime.com/t/duplicate-filter-with-more-conditions/90237/3): > > Hi @rds , your actual requirement appears to be that you are wanting to remove duplicates of Column 1 so that you retain only a single row for each product. Since you have specific criteria by which you want to choose which row to keep, you can just make use of the fact that by default the Duplicate Row Filter will keep the first row it encounters for each duplicate. You can change the criteria used by Duplicate Row Filter (especially by opening the “advanced” setting) but this won’t give you t… ---
**Author:** ![gonhaddock](https://forum-cdn.knime.com/user_avatar/forum.knime.com/gonhaddock/32/35425_2.png) [@gonhaddock](https://forum.knime.com/u/gonhaddock)\ **Post date:** [March 3, 2026, 1:55pm UTC](https://forum.knime.com/t/problems-deleting-duplicate-records/91536/4 "2026-03-03T13:55:07Z")
Hello @HolgerKleinert From my understanding what you are describing is a ‘GroupBy’() function with aggregate ‘first’ for those fields/columns that you don’t care. > **[GroupBy](https://hub.knime.com/knime/extensions/org.knime.features.base/latest/org.knime.base.node.preproc.groupby.GroupByNodeFactory)** > > Groups the rows of a table by the unique values in the selected group columns. A row is created for each unique set of values of the selected group column. The… BR ---
**Author:** ![HolgerKleinert](https://forum-cdn.knime.com/user_avatar/forum.knime.com/holgerkleinert/32/85044_2.png) [@HolgerKleinert](https://forum.knime.com/u/HolgerKleinert)\ **Post date:** [March 3, 2026, 2:07pm UTC](https://forum.knime.com/t/problems-deleting-duplicate-records/91536/5 "2026-03-03T14:07:51Z")
Hi Martin, thanks for the help! It works, but I have seen now that one “BELEGNUMMER” can have more positions with the same CPU-Date and -Time. One of them have 20 Positions and now 19 where lost. I think the Duplicate-Row-Filter is not the node I can use here ☹ BR, Holger ---
**Author:** ![mlauber71](https://forum-cdn.knime.com/user_avatar/forum.knime.com/mlauber71/32/73590_2.png) [@mlauber71](https://forum.knime.com/u/mlauber71)\ **Post date:** [March 3, 2026, 2:23pm UTC](https://forum.knime.com/t/problems-deleting-duplicate-records/91536/6 "2026-03-03T14:23:57Z")
@HolgerKleinert maybe you can provide a real world example with dummy data and explain what records you would want to keep and what the rules are for that. ---
**Author:** ![HolgerKleinert](https://forum-cdn.knime.com/user_avatar/forum.knime.com/holgerkleinert/32/85044_2.png) [@HolgerKleinert](https://forum.knime.com/u/HolgerKleinert)\ **Post date:** [March 3, 2026, 3:17pm UTC](https://forum.knime.com/t/problems-deleting-duplicate-records/91536/7 "2026-03-03T15:17:47Z")
Hi mlauber71, thanks for the tip! I’ve looked into it, and SQL seems a bit more complex. But I’d like to learn more, although I won’t have time until after my vacation in April. Until then, I’ll have to continue removing duplicates manually 😉 Regards, Holger ---
**Author:** ![ActionAndi](https://forum-cdn.knime.com/user_avatar/forum.knime.com/actionandi/32/76198_2.png) [@ActionAndi](https://forum.knime.com/u/ActionAndi)\ **Post date:** [March 4, 2026, 8:16am UTC](https://forum.knime.com/t/problems-deleting-duplicate-records/91536/8 "2026-03-04T08:16:22Z")
Hi Holger, in addition to @mlauber71 I highly recommend to provide some more context and data. The most efficient way would be to create a table with dummy data and show or describe precisely the targeted table. In: ![image](https://forum-cdn.knime.com/uploads/default/original/3X/7/b/7b5cc1dc90d273e237bb190476b7f3073972bd0a.png) Out: xyz Andi ---
**Author:** ![HolgerKleinert](https://forum-cdn.knime.com/user_avatar/forum.knime.com/holgerkleinert/32/85044_2.png) [@HolgerKleinert](https://forum.knime.com/u/HolgerKleinert)\ **Post date:** [March 4, 2026, 10:41am UTC](https://forum.knime.com/t/problems-deleting-duplicate-records/91536/9 "2026-03-04T10:41:32Z")
Hi Andi, As I described above, I “simply” need to delete duplicate records in the existing table before the next processing steps in Kinme. Or do you mean that I should create a new table with all the data but without the duplicate records using a filter? But what kind of filter can I use? One more thing: The table has 38 columns and, depending on how many transactions are in the source system, up to 5,000 rows, and these must not be grouped or modified. Regards, Holger ---
**Author:** ![HolgerKleinert](https://forum-cdn.knime.com/user_avatar/forum.knime.com/holgerkleinert/32/85044_2.png) [@HolgerKleinert](https://forum.knime.com/u/HolgerKleinert)\ **Post date:** [March 4, 2026, 11:50am UTC](https://forum.knime.com/t/problems-deleting-duplicate-records/91536/10 "2026-03-04T11:50:13Z")
Hi Gon, I’ve already tried using “group by”, but I can’t specify (for example): “DOCUMENTNUMBER” = “DOCUMENTNUMBER” and “CPU\_DATE” \<\> “CPU\_DATE” Or am I just being dense, or is there a special “group by” function in Knime? Regards, Holger ---
**Author:** ![mlauber71](https://forum-cdn.knime.com/user_avatar/forum.knime.com/mlauber71/32/73590_2.png) [@mlauber71](https://forum.knime.com/u/mlauber71)\ **Post date:** [March 4, 2026, 12:11pm UTC](https://forum.knime.com/t/problems-deleting-duplicate-records/91536/11 "2026-03-04T12:11:07Z")
@HolgerKleinert obviously you do not want to just delete duplicates because other conditions should apply. I think we currently do not understand these requirements. Maybe a clear example would make sense. Of course it is possible that we are not smart enough to understand your requirement but more often than not key to finding a solution is a clear definition of what should happen to which entry. ---
**Author:** ![ActionAndi](https://forum-cdn.knime.com/user_avatar/forum.knime.com/actionandi/32/76198_2.png) [@ActionAndi](https://forum.knime.com/u/ActionAndi)\ **Post date:** [March 4, 2026, 2:30pm UTC](https://forum.knime.com/t/problems-deleting-duplicate-records/91536/12 "2026-03-04T14:30:44Z")
Maybe Im lost in translation therefor I write now in german: Hi Holger, ich würde dir gerne helfen, jedoch verstehe ich nicht ganz was das Problem ist. Du hast eine Tabelle mit Duplikaten, die Du nach einer gewissen Regel entfernen möchtest. Die ersten Antworten in diesem Thread waren anscheinend nicht ausreichend, um das Problem zu lösen. Daher wäre es wichtig, wenn Du genau beschreibst, was deine Eingangsdaten sind und wie deine Ausgangsdaten aussehen sollen. Wenn es nur darum geht Duplikate zu entfernen, nimm den “Duplicate Row Filter”. Dort kannst Du auch einstellen, wie mit Duplikaten umgegangen werden soll: ![image](https://forum-cdn.knime.com/uploads/default/original/3X/f/3/f3f206c17f2b1872d5a7e2fd80137136859a873b.png) Andi ---
**Author:** ![Gergelysa1](https://forum-cdn.knime.com/letter_avatar_proxy/v4/letter/g/ccd318/32.png) [@Gergelysa1](https://forum.knime.com/u/Gergelysa1)\ **Post date:** [March 4, 2026, 6:57pm UTC](https://forum.knime.com/t/problems-deleting-duplicate-records/91536/13 "2026-03-04T18:57:13Z")
Use the Duplicate Row Filter node in KNIME, select BELEGNUMMER to remove duplicates, and it will keep one row per BELEGNUMMER—CPU\_DATE and CPU\_TIME don’t matter. ---
**Author:** ![HolgerKleinert](https://forum-cdn.knime.com/user_avatar/forum.knime.com/holgerkleinert/32/85044_2.png) [@HolgerKleinert](https://forum.knime.com/u/HolgerKleinert)\ **Post date:** [March 5, 2026, 7:58am UTC](https://forum.knime.com/t/problems-deleting-duplicate-records/91536/14 "2026-03-05T07:58:57Z")
[2602\_Doppelte\_Datensätze.xlsx](https://forum.knime.com/uploads/short-url/5wfPywRkDecdGIDWMYQNvkY4WBY.xlsx) (10.7 KB) ![image](https://forum-cdn.knime.com/uploads/default/original/3X/9/3/931270a826e56e375cbd1258d75210f7b5fcbdd5.png) Hallo Andi, danke dass ich dir das Problem in Deutsch erklären kann. Das macht es mir einfacher, da mein Englisch nicht so gut ist. Ich habe aus buchhalterischen Einzelposten z.B. diese zwei Fälle, die doppelt in meiner Knime-Verarbeitung stehen. Das kommt immer mal wieder vor und ich bekomme leider die IT-Schnittstelle für die Datenlieferung nicht angepasst, muss also in Knime eine maschinelle Lösung für das Problem finden. Ich schreibe mal etwas ausführlicher: Die BELEGNUMMER ist bei beiden gleich, nur bei beiden sind CPU\_DATE und CPU\_TIME unterschiedlich. Das wäre dann auch die (meine) Logik für das Löschen des doppelten Datensatzes. Also BELEGNUMMER ist identisch, aber wenn CPU\_DATE und CPU\_TIME unterschiedlich sind, lösche den doppelten Datensatz und behalte einen Datensatz. So wollte ich eigentlich vorgehen, aber ich bekomme das nicht hin. Es ist übrigens egal welcher Datensatz stehen bleibt, da für die Weiterverarbeitung die Info CPU\_DATE und CPU\_TIME nicht unbedingt nötig ist, da eine Identifizierung des Datensatzes im Quellsystem über die BELEGNUMMER stattfindet. Ich kann hier aber auch nicht einfach nach BELEGNUMMER gruppieren, da ein Beleg mehrere Zeilen haben kann, in meinem Prozess teilweise bis zu 200 Zeilen, und die dürfen nicht gelöscht bzw. zusammengefasst werden. Hier sind dann aber auch bei der BELEGNUMMER in den einzelnen Datensätzen CPU\_DATE und CPU\_TIME identisch. Derzeit mache ich mir in Knime mit einen Zwischenschritt einen Excel-Export, mache in Excel eine Pivot-Tabelle (Ergebnis als kleine Excel-Datei anbei), suche mir die Row-Nummer der doppelten Datensätze raus und lösche die Datensätze mit diesen Row-Nummern über einen „Rule-based Row Filter“. Leider ändern sich mit einer aktuellen Datenlieferung aus dem Quellsystem die Row-Nummern in Knime wieder, da neue Buchungen/Datensätze dazu kommen. Das zu automatisieren wäre klasse und ich hoffe Du kannst mir jetzt hier weiterhelfen. Noch eine Zusatzinfo: die Tabelle in Knime hat 38 Spalten und je nachdem wie viele Buchungen im Quellsystem vorhanden sind, bis zu 5.000 Zeilen. Diese dürfen nicht gruppiert oder verändert werden. Danke schon mal und Gruß Holger ---
**Author:** ![fe145f9fb2a1f6b](https://forum-cdn.knime.com/letter_avatar_proxy/v4/letter/f/f475e1/32.png) [@fe145f9fb2a1f6b](https://forum.knime.com/u/fe145f9fb2a1f6b)\ **Post date:** [March 5, 2026, 8:30am UTC](https://forum.knime.com/t/problems-deleting-duplicate-records/91536/15 "2026-03-05T08:30:04Z")
you will need BELEGNUMMER + BELEGPOSITION from your source system. Every other solution is guess work and will potentially result in an error in the short or long run. If you do not get the POSITION information, your best / only real option is: Groupby BELEGNUMMER + DATE + TIME, and aggregate: MAX (ANZAHL DATENSÄTZE), and COUNT(ANZAHL DATENSÄTZE). then Rule-based Row filter for MAX = COUNT. After that duplicate Row Filter on BELEGNUMMER + DATE + TIME. then join your original dataset (the full dataset) with the new table generated on BELEGNUMMER, DATE and TIME. that should leave you only those “rows” that have the same BELEGNUMMER, matching DATE and TIME, and are “complete” according to your ANZAHL DATENSÄTZE. BUT, if DATE or TIME varies within sets (think about 1 second difference in the source system), it will discard the whole dataset. tldr: get the BELEGPOSITION from your source system, everything else is irresponsible. ---
**Author:** ![ActionAndi](https://forum-cdn.knime.com/user_avatar/forum.knime.com/actionandi/32/76198_2.png) [@ActionAndi](https://forum.knime.com/u/ActionAndi)\ **Post date:** [March 5, 2026, 9:04am UTC](https://forum.knime.com/t/problems-deleting-duplicate-records/91536/16 "2026-03-05T09:04:24Z")
Also verstehe ich es richtig, dass Du Lösung 2 haben möchtest? ![image](https://forum-cdn.knime.com/uploads/default/original/3X/2/e/2ea9a7681cec21cefdea0517195863e7189e8c63.png) ---
**Author:** ![mlauber71](https://forum-cdn.knime.com/user_avatar/forum.knime.com/mlauber71/32/73590_2.png) [@mlauber71](https://forum.knime.com/u/mlauber71)\ **Post date:** [March 6, 2026, 8:30am UTC](https://forum.knime.com/t/problems-deleting-duplicate-records/91536/17 "2026-03-06T08:30:44Z")
@HolgerKleinert also ist es so dass die Dubletten entstehen aus den CPU Spalten weil davon nur eine Zeile stehen bleiben soll während die anderen Spalten die Informationen enthalten bestehen bleiben sollen. OK darüber kann ich nachdenken. Einfach Mal versucht die beiden CPU spalten auszuschließen und dann alle restlichen Spalten zu deduplizieren? Ein Beispiel mit mehr Zeilen kann evt die Komplexität besser darstellen. ![image](https://forum-cdn.knime.com/uploads/default/original/3X/8/8/888df440d4d7dc669ba21f92f0ccd9e7eb3c2d80.png) > **[Doppelte Datensätze - KNIME Forum (91536) – mlauber71](https://hub.knime.com/mlauber71/spaces/Forum/2026/Doppelte%20Datensa%CC%88tze%20-%20KNIME%20Forum%20(91536)~Y15Qr44Oq8s3pe0M/current-state)** > > Solutions for data science: find workflows, nodes and components, and collaborate in spaces. ---
**Author:** ![HolgerKleinert](https://forum-cdn.knime.com/user_avatar/forum.knime.com/holgerkleinert/32/85044_2.png) [@HolgerKleinert](https://forum.knime.com/u/HolgerKleinert)\ **Post date:** [March 6, 2026, 10:36am UTC](https://forum.knime.com/t/problems-deleting-duplicate-records/91536/18 "2026-03-06T10:36:28Z")
Hi Martin, nein ich brauche Lösung 1. Gruß Holger ---
**Author:** ![ActionAndi](https://forum-cdn.knime.com/user_avatar/forum.knime.com/actionandi/32/76198_2.png) [@ActionAndi](https://forum.knime.com/u/ActionAndi)\ **Post date:** [March 6, 2026, 5:08pm UTC](https://forum.knime.com/t/problems-deleting-duplicate-records/91536/19 "2026-03-06T17:08:42Z")
Na dann ist es doch ganz einfach: ![image](https://forum-cdn.knime.com/uploads/default/original/3X/1/2/12c5a8418aea49e90025b3a6e400411f2bb92514.png) ---
**Author:** ![HolgerKleinert](https://forum-cdn.knime.com/user_avatar/forum.knime.com/holgerkleinert/32/85044_2.png) [@HolgerKleinert](https://forum.knime.com/u/HolgerKleinert)\ **Post date:** [March 9, 2026, 6:14am UTC](https://forum.knime.com/t/problems-deleting-duplicate-records/91536/20 "2026-03-09T06:14:46Z")
Vielen Dank! Ich hätte nicht gedacht dass die Lösung so einfach ist. Man muss es nur wissen 😉 Ich mag diese Community, in der immer eine Problemlösung gefunden wird. Gruß Holger [Next page](https://forum.knime.com/t/problems-deleting-duplicate-records/91536.md?page=2)