FUZZY MATCH ALTERNATIVES

[jmanuelml21]

GD, Forum Users

I got the following problem; hopefully as you’re the forum-analytic gods you will be merciful with this poor guy’s problem.
I got a big big list of string inputs (1000k+). These inputs should have been standarized and only should have been input only when chosen from a 1k options list but things did not go as planned and data is now a mess; writing each input with different characters, random digits, missing letters, words in different order, etc…

I need to narrow down the original 1k option list.
At first, thought fuzzy match would be my solution but i need the “dictionary”, or the 1k list so it can match the different inputs.
Now i think i need to string manipulate and get the 1k option list manually, checking the inputs 1 by 1. So far i’ve tried string manipulation to remove digits and specific words, groupby node but still, my selection is way too big to do it manually.
screenshot

COULD YOU PLEASE HELP ME!?!?
How would you approach this problem?

Many thanks in advance, guys!

[goodvirus]

Hi, have you tried the String similarty node?

[mlauber71]

@jmanuelml21 here is an example of how you could identify duplicates and bring them together without having a ‘ground truth’. The workflow will try to determine the groups for itself.

How can I define and list the duplication in an adress data set sucessfully, with using String Distances node and Similarity Search node?

@DAmei I had another look and I built a slightly over engineered workflow to try and get what you want combining some of the approached already mentioned. In your case there might not be a ground truth against wich to match but rather a lot of possible combinations. So configuration will be necessary and maybe exclude some obvious detaches in the first place. What the workflow does: the address is turned into a standardised string. The composition of the string will influence the matching. So…

How can I define and list the duplication in an adress data set sucessfully, with using String Distances node and Similarity Search node?

@DAmei you could use the slider to set a value between 0 and 100 - the lower the more strict the definition of a duplicate gets. In your example data there is a seeming identical value while the name is very long in the field. You will have to experiment with the settings. [image] And also you might think about limiting the number of characters you would use to construct the company name. If you want to stress the importance of a component you could use that a few times, to give it more weigh…

Then you can do more address matching and deduplication with these examples:

address deduplication, string similarity and fingerprinting (a collection) –...

[izaychik63]

@jmanuelml21 , look at this trail:

Renaming a string in a row based on another column

Hello all, i am working on a small project at school. Actually i am at one point where i have no clue how to go further. I have a column with company names where due to spelling mistakes the company name can be spelled wrong. For example the company name has 2 instead of 1 space, a letter is missing or the dot notation is wrong. In this case i have done a count on the company names and the one with the most hits is the right name. I would like to change all the other columns that are similiar …

The approach somewhat simpler. If you prefer different metrics just connect String Distance to Similarity Search node.

You can connect WF from the trail above.
The idea to compare same lists.

[jmanuelml21]

You mean “similarity search”?
Please let me know.

Thanks!!

[izaychik63]

See here

String Similarity — NodePit

[jmanuelml21]

many thanks for your input. This is greatly appreciated. I’m giving your workflow a try but looks like it’s not as simple as i thought. I will have to go over calmly and understand each step as i go over it. There are many resources which i’m not familiar with. I’lL let you know how it goes.
again, thanks!