Making Rules using ML for family ID creation

[pujappathak]

Hi,

I have data which contains demographic information of candidates like first name, last name, father’s name, mother’s name, pincode, etc. I want to group these candidates over certain rules so as to find out which 2/3/multiple candidates belong to same family. I tried simply grouping them by last name and father’s name OR last name and pincode. Can there be a better approach?

[mlauber71]

@pujappathak first question would be: what would constitute a “family” and will your data contain the information to (possibly) get you there. Is it the name? Depending on the size and frequency of a name this might be a challenge. Then there could be an address and (maybe) a (last) name. Do you know the name of the father in a single data set (is the information reliable?).

Here is an example of automatically grouping similar addresses without a ‘ground truth’ - but the question would be would it be suitable for your task. Probably not:

How can I define and list the duplication in an adress data set sucessfully, with using String Distances node and Similarity Search node?

@DAmei I had another look and I built a slightly over engineered workflow to try and get what you want combining some of the approached already mentioned. In your case there might not be a ground truth against wich to match but rather a lot of possible combinations. So configuration will be necessary and maybe exclude some obvious detaches in the first place. What the workflow does: the address is turned into a standardised string. The composition of the string will influence the matching. So…

How can I define and list the duplication in an adress data set sucessfully, with using String Distances node and Similarity Search node?

@DAmei you could use the slider to set a value between 0 and 100 - the lower the more strict the definition of a duplicate gets. In your example data there is a seeming identical value while the name is very long in the field. You will have to experiment with the settings. [image] And also you might think about limiting the number of characters you would use to construct the company name. If you want to stress the importance of a component you could use that a few times, to give it more weigh…

For further inspirations about addresses and fingerprinting and so on you might want to explore this collection (not saying you will find an answer without answering the questions mentioned):

address deduplication, string similarity and fingerprinting (a collection) –...