Check similar or duplicate content using text analytics

[omprakashjena]

Hi,

I have a dataset that contains some text (comments) in a column. I want to check if there is any duplicate or similar content in that column. Similar or duplicate content can be determined where the meaning is similar but the structure of the sentence is different.
A sample workflow would be really helpful.

[HansS]

Hi @omprakashjena

@omprakashjena

A sample workflow would be really helpful.

In that case, a sample dataset would be helpfull. What is your input and what is the output that you expect. The logic in your post is a little bit fuzzy to me.

gr. Hans

[mlauber71]

@omprakashjena you could check this example. It has been used to bundle together similar addresses but it might also be adapted for other tasks. It does not need a ground truth but would start to combine similar strings:

How can I define and list the duplication in an adress data set sucessfully, with using String Distances node and Similarity Search node?

@DAmei I had another look and I built a slightly over engineered workflow to try and get what you want combining some of the approached already mentioned. In your case there might not be a ground truth against wich to match but rather a lot of possible combinations. So configuration will be necessary and maybe exclude some obvious detaches in the first place. What the workflow does: the address is turned into a standardised string. The composition of the string will influence the matching. So…

You could also explore more examples about string matching:

address deduplication, string similarity and fingerprinting (a collection) –...