Perplexity

[monica_rosciano]

Hello,
I have applied the topic modeling using Topic Extractor (parallel LDA) node.
Now I would like to evaluate the results by calculating the Perplexity. How can Ido it in knime.
Thank you,
Monica

[thor_landstrom]

Hey @monica_rosciano,

I would take a look at the following thread below:

How to interpret perplexity in topic modeling?

Hello everyone, I would like to find out the optimal topic number by using the two-step perplexity method used in this workflow (“Block 2”): Yet, I am not sure how to interpret the resulting charts correctly. Here is an example of how I would proceed: Step 1 - wide topic range (2 to 80) [grafik] → the “elbow” range is between 6 and 10 Step 2 - narrowing down the topic range (6 to 10) [grafik] Now, acording to the last chart, what is the optimal number (7, 8 or 9)? Is my approach cor…

It also has a sample workflow in there where they calculate perplexity, which you can possibly use to adapt to your use case.

Hope this helps,
TL

[monica_rosciano]

Hi, Thor,
thank you for your answer, but I need to know how to calculate and plot Perplexity in knime, not its interpretation.
Regards,
Monica

[thor_landstrom]

Have you had the chance to take a look at the workflow the user mentions?

Topic Models from Reviews – knime

There is a component that calculates perplexity. I would also take a look at this workflow on the community hub:

Topic Models from Reviews – mgfau

Hope this helps,
TL