Hi, seeking an advise from expert on how to do similarity search. I want a function where user can search anything, if it has the same meaning it will show same answer.
Example.
Set 1: What are some popular tourist attractions in Paris, France?
Set 2: Which famous landmarks should I visit when traveling to Paris, France?
Answer: Paris, the City of Light, is home to numerous iconic landmarks and attractions. Some of the most popular tourist attractions include the Eiffel Tower, the Louvre Museum, Notre-Dame Cathedral, Arc de Triomphe, and the Champs-Élysées. Visitors can also explore the Montmartre neighborhood, take a Seine River cruise, or visit the Palace of Versailles.
to be honest I don’t think that is what the Table View is capable of by default.
When you say you want to search for “meaning” it sounds somewhat like a potential use case for an App / Component that leverages KNIME’s Generative AI integration:
From a high-level point of view this would require:
Using embedding models to vectorize your data and storing the vector / data pairs in a vectorstore
creating a user interface using widgets / component composite views to obtain user input. User input is then also vectorized and a vector search is performed on the data in the vectorstore
the “most similar” search results (to the user input) are obtained and displayed to the user (that could be what is displayed in a Table View
As you can probably guess this is not a “light-weight, fast-to-realise” topic…
To get you started I recommend to install that AI extension and to check out the LLM-related examples on the KNIME hub:
Also @mlauber71 wrote an article on embeddings etc. including KNIME:
@WanRashid you might want to check some examples of how to have a dialogue in a knime component with you own data using local LLMs in this collection. I assume that what you mean by similarity search could be what a vector store does.
And the accompanying articles:
If you are fine with also using OpenAI you can check the KNIME space:
