Text Processing: TF Node counts incorrectly?

[mwiegand]

Hi,

maybe I make a mistake as I am not that familiar with the text processing nodes but when providing some aid in:

Counting words in a text string

Hello, it’s a pleasure tHello, it’s a pleasure to greet you. I need to create a new column that gives me the result of how many words a name contains. Example: Juan Antonio Zavala Perez has 4 words. Thank you very much in advance Regards.

I happen to notice a way to high count of words using the TF Node. To be precise, the amount is twice as high.

TF

Edit: Here is the test workflow

79614 - Count Words – mwiegand

Best
Mike

[k10shetty1]

Hi @mwiegand,

Thanks for bringing this up.

You’ve likely assigned the same column to both ‘Title’ and ‘Full Text’ column in the Strings to Document node, The TF node counts words in both these fields. To double check, use the Document Viewer node.

To fix this, either create an empty column named “fulltext” and assign it to Full Text in the Strings to Document node. Or, use the Math node to divide the term frequency by 2.

Best,
Keerthan

[mwiegand]

Good morning @k10shetty1,

you are right but doesn’t this beg the question if either the nodes description is wrong or its behavior:

Computes the relative term frequency (tf) of each term according to each document and adds

TF

Apparently the title belongs to the document so maybe a good compromise is to add the ability in the TF node to select to count the whole document or aspects of it:

1. Title
2. Text
3. Meta: Whole, Source, Categories, Authors

Interestingly, the meta information is not taken into account by the TF node currently.

I updated the test workflow for reference in case anyone stumbles across that subject.

Best
Mike