Maybe you take a look at this article dealing with duplicates. The other link would give you an example how to use Window functions like rank to control what would count as a duplicate.
Dealing with duplicates is a constant theme with data scientist. And a lot of things can go wrong. The easiest ways to deal with them is SQL’s GROUP BY or DISTINCT. Just get rid of them and be done. But as this examples might demonstrate this might not always be the best option. Even if your data provider swears your combined IDs are unique especially in Big Data scenarios there might still be lurking some muddy duplicates and you should still be able to deal with them.
And you should be able t…
MySQL also does support these functions:
https://dev.mysql.com/doc/refman/8.0/en/window-function-descriptions.html
1 Like